Nov 01 12:21:36.609829 ip-10-0-140-168 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 12:21:36.984547 ip-10-0-140-168 kubenswrapper[2583]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:21:36.984547 ip-10-0-140-168 kubenswrapper[2583]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 12:21:36.984547 ip-10-0-140-168 kubenswrapper[2583]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:21:36.984547 ip-10-0-140-168 kubenswrapper[2583]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:21:36.988412 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988333    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:36.988412 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988413    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988418    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988421    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988425    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988427    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988430    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988433    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988436    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988439    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988441    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988444    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988446    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988450    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988452    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988455    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988457    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988461    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988463    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988466    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988469    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:36.988480 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988471    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988474    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988477    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988480    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988482    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988485    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988488    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988491    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988494    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988497    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988499    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988503    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988505    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988508    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988510    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988525    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988528    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988530    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988533    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988535    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:36.988957 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988538    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988540    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988543    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988546    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988548    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988551    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988553    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988556    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988560    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988565    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988568    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988571    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988573    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988576    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988579    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988582    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988585    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988588    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988590    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:36.989429 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988593    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988596    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988598    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988601    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988603    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988606    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988608    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988611    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988613    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988616    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988618    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988621    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988625    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988628    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988632    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988635    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988638    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988640    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988643    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:36.989892 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988646    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988648    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988651    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988653    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988656    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988658    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988661    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988663    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988667    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988670    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988672    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988675    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988677    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988680    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988682    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988685    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988689    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988692    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988694    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:36.990336 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988697    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988699    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988702    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988705    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988707    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988710    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988712    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.988715    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989548    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989556    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989559    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989562    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989564    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989567    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989570    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989572    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989575    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989577    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:36.990844 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989580    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989584    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989589    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989593    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989595    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989598    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989601    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989604    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989606    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989609    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989611    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989614    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989616    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989619    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989622    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989624    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989627    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989629    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989632    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989634    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:36.991293 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989637    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989640    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989643    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989646    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989648    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989651    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989653    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989656    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989658    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989661    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989663    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989665    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989668    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989670    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989673    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989677    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989679    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989682    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989685    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:36.991781 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989687    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989690    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989693    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989695    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989697    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989700    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989703    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989705    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989708    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989710    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989713    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989715    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989718    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989720    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989722    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989725    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989729    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989731    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989734    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989736    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:36.992234 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989739    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989741    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989743    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989746    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989749    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989751    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989753    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989756    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989760    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989762    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989765    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989768    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989771    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989773    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989776    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989778    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989781    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989783    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989786    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989789    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:36.992711 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989791    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989794    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989796    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989799    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989801    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989804    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989806    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989809    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989813    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989817    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989820    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989823    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989826    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989829    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989832    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989834    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.989838    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990346    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990351    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:36.993197 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990355    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990358    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990361    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990364    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990366    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990370    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990374    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990377    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990380    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990383    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990385    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990388    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990390    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990393    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990396    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990398    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990401    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990403    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990406    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:36.993662 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990408    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990411    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990413    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990416    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990419    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990421    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990424    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990427    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990429    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990432    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990434    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990437    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990439    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990442    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990445    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990447    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990450    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990453    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990455    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990458    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:36.994098 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990460    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990463    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990465    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990467    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990470    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990473    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990475    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990478    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990481    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990483    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990486    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990488    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990491    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990493    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990496    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990498    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990501    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990504    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990506    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990509    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:36.994664 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990524    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990527    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990529    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990532    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990534    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990537    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990540    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990543    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990545    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990548    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990551    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990553    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990556    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990559    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990561    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990564    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990566    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990569    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990571    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:36.995176 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990574    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990578    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990581    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990584    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990586    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990589    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990591    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990594    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990596    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990599    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990605    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990610    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990613    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990615    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990618    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990620    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990623    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990626    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990628    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:36.995639 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990631    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990633    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990636    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990638    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990641    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990645    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.990647    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992173    2583 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992184    2583 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992191    2583 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992196    2583 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992201    2583 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992204    2583 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992209    2583 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992213    2583 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992216    2583 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992219    2583 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992223    2583 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992226    2583 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992230    2583 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992233    2583 flags.go:64] FLAG: --cgroup-root=""
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992236    2583 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 12:21:36.996084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992239    2583 flags.go:64] FLAG: --client-ca-file=""
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992242    2583 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992247    2583 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992251    2583 flags.go:64] FLAG: --cluster-domain=""
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992255    2583 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992258    2583 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992261    2583 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992264    2583 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992269    2583 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992272    2583 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992275    2583 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992278    2583 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992281    2583 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992284    2583 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992289    2583 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992293    2583 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992296    2583 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992300    2583 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992303    2583 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992306    2583 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992309    2583 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992312    2583 flags.go:64] FLAG: --enable-server="true"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992315    2583 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992319    2583 flags.go:64] FLAG: --event-burst="100"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992322    2583 flags.go:64] FLAG: --event-qps="50"
Nov 01 12:21:36.996614 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992326    2583 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992329    2583 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992332    2583 flags.go:64] FLAG: --eviction-hard=""
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992336    2583 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992339    2583 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992342    2583 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992345    2583 flags.go:64] FLAG: --eviction-soft=""
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992348    2583 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992351    2583 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992355    2583 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992358    2583 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992361    2583 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992364    2583 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992367    2583 flags.go:64] FLAG: --feature-gates=""
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992372    2583 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992375    2583 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992379    2583 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992382    2583 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992385    2583 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992389    2583 flags.go:64] FLAG: --help="false"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992392    2583 flags.go:64] FLAG: --hostname-override="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992395    2583 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992398    2583 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 12:21:36.997283 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992401    2583 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992404    2583 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992408    2583 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992411    2583 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992414    2583 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992417    2583 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992420    2583 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992423    2583 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992426    2583 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992429    2583 flags.go:64] FLAG: --kube-reserved=""
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992432    2583 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992434    2583 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992437    2583 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992440    2583 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992443    2583 flags.go:64] FLAG: --lock-file=""
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992446    2583 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992449    2583 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992452    2583 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992458    2583 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992461    2583 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992464    2583 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992467    2583 flags.go:64] FLAG: --logging-format="text"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992470    2583 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992473    2583 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 12:21:36.997860 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992476    2583 flags.go:64] FLAG: --manifest-url=""
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992479    2583 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992484    2583 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992487    2583 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992491    2583 flags.go:64] FLAG: --max-pods="110"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992494    2583 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992497    2583 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992500    2583 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992503    2583 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992506    2583 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992509    2583 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992521    2583 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992529    2583 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992532    2583 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992535    2583 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992539    2583 flags.go:64] FLAG: --pod-cidr=""
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992542    2583 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992545    2583 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992548    2583 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992550    2583 flags.go:64] FLAG: --port="10250"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992553    2583 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992557    2583 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0098837b9375d32bb"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992560    2583 flags.go:64] FLAG: --qos-reserved=""
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992563    2583 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992566    2583 flags.go:64] FLAG: --register-node="true"
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992569    2583 flags.go:64] FLAG: --register-with-taints=""
Nov 01 12:21:36.999208 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992573    2583 flags.go:64] FLAG: --registry-burst="10"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992576    2583 flags.go:64] FLAG: --registry-qps="5"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992579    2583 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992582    2583 flags.go:64] FLAG: --reserved-memory=""
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992587    2583 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992590    2583 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992596    2583 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992599    2583 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992602    2583 flags.go:64] FLAG: --runonce="false"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992605    2583 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992608    2583 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992611    2583 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992614    2583 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992617    2583 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992620    2583 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992623    2583 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992626    2583 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992629    2583 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992632    2583 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992635    2583 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992638    2583 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992641    2583 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992644    2583 flags.go:64] FLAG: --system-cgroups=""
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992647    2583 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992652    2583 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 12:21:36.999873 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992655    2583 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992658    2583 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992663    2583 flags.go:64] FLAG: --tls-min-version=""
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992666    2583 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992669    2583 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992672    2583 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992674    2583 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992679    2583 flags.go:64] FLAG: --v="2"
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992683    2583 flags.go:64] FLAG: --version="false"
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992687    2583 flags.go:64] FLAG: --vmodule=""
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992691    2583 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.992695    2583 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992876    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992881    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992885    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992888    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992890    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992893    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992896    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992898    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992901    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992904    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:37.000486 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992906    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992909    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992912    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992914    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992917    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992919    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992922    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992925    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992927    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992930    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992932    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992935    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992937    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992941    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992945    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992948    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992952    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992956    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992958    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:37.001763 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992961    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992963    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992966    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992969    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992973    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992975    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992978    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992980    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992983    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992985    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992988    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992990    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992993    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992995    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.992998    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993000    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993003    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993005    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993008    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:37.002235 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993010    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993013    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993016    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993018    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993021    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993023    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993026    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993028    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993030    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993033    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993036    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993039    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993042    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993045    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993047    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993049    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993052    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993056    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993059    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:37.002694 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993061    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993064    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993067    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993069    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993072    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993074    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993077    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993079    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993082    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993084    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993087    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993089    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993093    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993096    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993099    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993102    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993106    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993109    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993112    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:37.003161 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993114    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993116    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993119    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993121    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993124    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993127    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993130    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993133    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993135    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993138    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993140    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993144    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993146    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993149    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993152    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993154    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993157    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993159    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993162    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:37.003624 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993165    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.993190    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993225    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993230    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993234    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993237    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993240    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993242    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993245    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993248    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993250    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993254    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993256    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993259    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993261    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993264    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:37.004078 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993266    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993269    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993274    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993278    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993281    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993284    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993287    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993289    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993292    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993295    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993298    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993301    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993304    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993306    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993309    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993311    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993313    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993316    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993318    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:37.004462 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993321    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993323    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993326    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993328    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993331    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993333    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993336    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993338    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993341    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993343    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993346    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993349    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993351    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993354    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993356    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993360    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993362    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993365    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993367    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:37.004923 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993370    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993372    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993376    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993379    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993381    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993384    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993387    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993389    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993392    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993394    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993397    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993399    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993402    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993404    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993407    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993409    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993411    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993414    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993417    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:37.005375 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993419    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993422    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993426    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993430    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993433    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993435    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993438    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993440    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993443    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993446    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993449    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993451    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993454    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993456    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993459    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993462    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993464    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993467    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993469    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:37.005882 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993472    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993475    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993477    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993480    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993482    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993484    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993487    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993490    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993492    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993494    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993497    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993500    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993504    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993506    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993509    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:36.993511    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.999767    2583 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:36.999782    2583 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.000660    2583 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.000667    2583 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 12:21:37.006325 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000724    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000729    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000733    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000737    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000740    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000743    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000746    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000749    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000752    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000754    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000757    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000760    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000762    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000765    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000768    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000770    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000773    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000775    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000778    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:37.006819 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000780    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000783    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000786    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000789    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000791    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000794    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000796    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000799    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000802    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000804    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000808    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000811    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000814    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000816    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000819    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000821    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000824    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000826    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000828    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000831    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:37.007360 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000834    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000836    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000839    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000841    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000844    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000846    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000849    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000851    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000854    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000856    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000861    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000864    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000868    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000871    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000873    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000876    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000878    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000881    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000884    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:37.008142 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000887    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000889    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000893    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000898    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000901    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000904    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000907    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000909    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000912    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000915    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000918    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000920    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000923    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000925    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000928    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000930    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000933    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000935    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000938    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:37.008799 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000940    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000943    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000946    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000948    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000950    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000953    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000956    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000958    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000961    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000964    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000966    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000969    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000972    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000974    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000977    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000980    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000982    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000985    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000988    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000991    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:37.009246 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000993    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000996    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.000999    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001001    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001004    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001007    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001009    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001012    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001014    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.001041    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001185    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001189    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001192    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001195    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001198    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001200    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:37.009734 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001203    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001205    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001208    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001211    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001213    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001216    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001218    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001221    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001224    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001226    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001229    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001232    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001234    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001237    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001239    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001242    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001245    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001248    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001250    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001253    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:37.010112 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001255    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001258    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001261    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001263    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001266    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001268    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001271    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001274    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001276    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001279    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001281    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001284    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001287    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001289    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001292    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001294    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001297    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001299    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001302    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:37.010756 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001305    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001307    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001310    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001313    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001315    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001318    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001322    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001326    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001329    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001333    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001336    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001338    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001341    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001343    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001346    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001348    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001351    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001353    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001357    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:37.011231 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001360    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001363    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001366    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001368    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001371    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001373    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001376    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001378    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001381    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001383    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001386    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001388    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001391    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001393    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001395    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001398    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001401    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001403    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001406    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001408    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:37.011695 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001411    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001413    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001416    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001424    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001427    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001430    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001432    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001435    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001438    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001441    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001443    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001445    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001448    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001451    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001454    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001456    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001459    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001461    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001464    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001467    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:37.012168 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001469    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001472    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.001498    2583 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001542    2583 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001547    2583 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001550    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001553    2583 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001557    2583 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001561    2583 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001564    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001568    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001570    2583 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001573    2583 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001576    2583 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001578    2583 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:21:37.012661 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001581    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001584    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001586    2583 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001589    2583 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001593    2583 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001596    2583 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001599    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001602    2583 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001604    2583 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001608    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001610    2583 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001613    2583 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001615    2583 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001618    2583 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001620    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001623    2583 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001625    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001628    2583 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001630    2583 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:21:37.013032 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001634    2583 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001637    2583 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001639    2583 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001641    2583 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001644    2583 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001646    2583 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001649    2583 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001651    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001654    2583 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001656    2583 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001659    2583 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001662    2583 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001664    2583 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001666    2583 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001669    2583 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001672    2583 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001675    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001677    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001679    2583 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001682    2583 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:21:37.013472 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001684    2583 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001687    2583 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001689    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001692    2583 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001694    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001697    2583 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001700    2583 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001702    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001704    2583 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001707    2583 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001709    2583 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001712    2583 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001716    2583 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001718    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001721    2583 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001723    2583 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001725    2583 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001728    2583 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001731    2583 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001733    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:21:37.013947 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001735    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001738    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001741    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001743    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001746    2583 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001748    2583 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001751    2583 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001753    2583 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001756    2583 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001758    2583 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001761    2583 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001763    2583 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001766    2583 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001769    2583 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001772    2583 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001774    2583 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001778    2583 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001780    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001783    2583 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001785    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:21:37.014449 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001788    2583 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001791    2583 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001793    2583 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001796    2583 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001799    2583 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001801    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001804    2583 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001806    2583 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001810    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001813    2583 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001816    2583 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001819    2583 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001821    2583 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001824    2583 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:37.001826    2583 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.001877    2583 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.003752    2583 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.004652    2583 server.go:1014] "Starting client certificate rotation"
Nov 01 12:21:37.014927 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.004750    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 12:21:37.015347 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.005492    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 12:21:37.015347 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.006907    2583 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 12:21:37.015347 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.010017    2583 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 12:21:37.022395 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.022378    2583 log.go:25] "Validated CRI v1 runtime API"
Nov 01 12:21:37.025464 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.025450    2583 log.go:25] "Validated CRI v1 image API"
Nov 01 12:21:37.027538 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.026680    2583 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 12:21:37.029073 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.029052    2583 fs.go:135] Filesystem UUIDs: map[4007a117-6838-4a92-ba64-07d5882b1f0a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a6560e3c-2503-4125-842e-43411efd96ab:/dev/nvme0n1p3]
Nov 01 12:21:37.029137 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.029069    2583 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 01 12:21:37.034087 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.033978    2583 manager.go:217] Machine: {Timestamp:2026-11-01 12:21:37.032655548 +0000 UTC m=+0.312583096 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100783 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22eb1636ee97c907452bcfaf8f6d15 SystemUUID:ec22eb16-36ee-97c9-0745-2bcfaf8f6d15 BootID:981d3d92-f884-41ad-a9cd-22b9a28ef5f9 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:f6:30:d6:cd Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:f6:30:d6:cd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:42:af:11:df:1f:a0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 12:21:37.034542 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.034532    2583 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 12:21:37.034626 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.034609    2583 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 12:21:37.037103 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.037081    2583 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 12:21:37.037262 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.037106    2583 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-168.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 12:21:37.037306 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.037265    2583 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 12:21:37.037306 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.037273    2583 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 12:21:37.037356 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.037308    2583 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 12:21:37.037356 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.037325    2583 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 12:21:37.037997 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.037987    2583 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 12:21:37.041606 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.041594    2583 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 12:21:37.041726 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.041717    2583 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 12:21:37.042392 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.042384    2583 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 12:21:37.042421 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.042397    2583 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 12:21:37.042421 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.042408    2583 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 12:21:37.042421 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.042417    2583 kubelet.go:405] "Adding apiserver pod source"
Nov 01 12:21:37.042510 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.042427    2583 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 12:21:37.042852 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.042835    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 12:21:37.044964 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.044947    2583 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 12:21:37.049790 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.049777    2583 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 12:21:37.051964 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.051928    2583 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 12:21:37.052043 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.051993    2583 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 12:21:37.052201 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.052187    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 12:21:37.052255 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.052211    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 12:21:37.052255 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.052222    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 12:21:37.052255 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.052233    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 12:21:37.052255 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.052248    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 12:21:37.052396 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.052258    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 12:21:37.053212 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.053197    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 12:21:37.053285 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.053225    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 12:21:37.053285 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.053247    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 12:21:37.053285 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.053256    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 12:21:37.053285 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.053270    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 12:21:37.053285 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.053285    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 12:21:37.053821 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.053811    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 12:21:37.053870 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.053825    2583 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 12:21:37.057383 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.057368    2583 server.go:1263] "Started kubelet"
Nov 01 12:21:37.057466 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.057418    2583 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 12:21:37.057534 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.057454    2583 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 12:21:37.057534 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.057511    2583 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 12:21:37.058104 ip-10-0-140-168 systemd[1]: Started Kubernetes Kubelet.
Nov 01 12:21:37.058573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.058550    2583 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 12:21:37.060223 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.060208    2583 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 12:21:37.062851 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.062831    2583 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 12:21:37.063329 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.063313    2583 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 12:21:37.064174 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064161    2583 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 12:21:37.064332 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064313    2583 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 12:21:37.064421 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064401    2583 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 12:21:37.064473 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064446    2583 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 12:21:37.064473 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064455    2583 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 12:21:37.064610 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064531    2583 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 12:21:37.064610 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064546    2583 factory.go:55] Registering systemd factory
Nov 01 12:21:37.064610 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064556    2583 factory.go:223] Registration of the systemd container factory successfully
Nov 01 12:21:37.064824 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064808    2583 factory.go:153] Registering CRI-O factory
Nov 01 12:21:37.064879 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064827    2583 factory.go:223] Registration of the crio container factory successfully
Nov 01 12:21:37.064879 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.064820    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:37.064879 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064847    2583 factory.go:103] Registering Raw factory
Nov 01 12:21:37.064879 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.064863    2583 manager.go:1199] Started watching for new ooms in manager
Nov 01 12:21:37.065360 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.065337    2583 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 01 12:21:37.065777 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.065757    2583 manager.go:319] Starting recovery of all containers
Nov 01 12:21:37.076293 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.076277    2583 manager.go:324] Recovery completed
Nov 01 12:21:37.082583 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.082539    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:37.084575 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.084545    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:37.084650 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.084592    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:37.084650 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.084608    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:37.085128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.085114    2583 cpu_manager.go:226] "Starting" policy="none"
Nov 01 12:21:37.085128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.085120    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:37.085128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.085124    2583 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 12:21:37.085269 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.085139    2583 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 12:21:37.085838 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.085824    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:37.085886 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.085850    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:37.085886 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.085862    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:37.087480 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.087460    2583 policy_none.go:50] "Start"
Nov 01 12:21:37.087480 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.087479    2583 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 12:21:37.087632 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.087494    2583 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 12:21:37.165084 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.165054    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:37.194252 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.194225    2583 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-140-168.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 01 12:21:37.194252 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.194227    2583 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-140-168.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 01 12:21:37.229758 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.229730    2583 policy_none.go:44] "Start"
Nov 01 12:21:37.255019 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.239299    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-168.us-west-2.compute.internal.18e3eb36c315de04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-168.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-140-168.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 12:21:37.057340932 +0000 UTC m=+0.337268479,LastTimestamp:2026-11-01 12:21:37.057340932 +0000 UTC m=+0.337268479,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-168.us-west-2.compute.internal,}"
Nov 01 12:21:37.265314 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.265175    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:37.265770 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.265743    2583 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 12:21:37.267999 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.267985    2583 manager.go:341] "Starting Device Plugin manager"
Nov 01 12:21:37.268079 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.268009    2583 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 12:21:37.268079 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.268018    2583 server.go:87] "Starting device plugin registration server"
Nov 01 12:21:37.268245 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.268232    2583 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 12:21:37.268307 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.268245    2583 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 12:21:37.268359 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.268327    2583 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 12:21:37.268405 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.268398    2583 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 12:21:37.268455 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.268406    2583 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 12:21:37.268923 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.268906    2583 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 12:21:37.269005 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.268946    2583 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:37.281375 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.281336    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 12:21:37.282608 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.282590    2583 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 12:21:37.282608 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.282611    2583 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 12:21:37.282747 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.282626    2583 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 12:21:37.282747 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.282663    2583 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 01 12:21:37.292619 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.292602    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 12:21:37.292712 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.292659    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-g8rng"
Nov 01 12:21:37.340040 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.340013    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 01 12:21:37.340040 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.340012    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-140-168.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 01 12:21:37.340183 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.340015    2583 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 01 12:21:37.340183 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.340016    2583 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-140-168.us-west-2.compute.internal.18e3eb36c4b56879  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-168.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-168.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-168.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 12:21:37.084573817 +0000 UTC m=+0.364501573,LastTimestamp:2026-11-01 12:21:37.084573817 +0000 UTC m=+0.364501573,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-168.us-west-2.compute.internal,}"
Nov 01 12:21:37.369037 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.369013    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:37.369840 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.369824    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:37.369926 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.369857    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:37.369926 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.369873    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:37.369926 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.369904    2583 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.383016 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.382994    2583 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"]
Nov 01 12:21:37.383083 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.383050    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:37.383808 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.383788    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:37.383883 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.383819    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:37.383883 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.383829    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:37.385960 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.385948    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:37.386108 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.386092    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.386176 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.386165    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:37.386616 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.386591    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:37.386703 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.386626    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:37.386703 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.386661    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:37.386703 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.386690    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:37.386796 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.386709    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:37.386796 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.386719    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:37.388686 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.388673    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.388732 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.388696    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:37.389365 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.389351    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:37.389435 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.389379    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:37.389435 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.389392    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:37.396265 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.396252    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:37.415077 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.415058    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found" node="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.417831 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.417814    2583 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found" node="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.426630 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.426615    2583 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.426723 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.426638    2583 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-140-168.us-west-2.compute.internal\": node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:37.466069 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.466049    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7042c7ca4bb39d530e5b25fb56952b15-config\") pod \"kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"7042c7ca4bb39d530e5b25fb56952b15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.466069 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.466074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.466218 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.466092    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.548894 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.548818    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:37.567102 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.567077    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.567169 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.567108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.567169 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.567127    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7042c7ca4bb39d530e5b25fb56952b15-config\") pod \"kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"7042c7ca4bb39d530e5b25fb56952b15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.567233 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.567176    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.567274 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.567229    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7042c7ca4bb39d530e5b25fb56952b15-config\") pod \"kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"7042c7ca4bb39d530e5b25fb56952b15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.567274 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.567264    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.649109 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.649078    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:37.717581 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.717554    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.720588 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:37.720571    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:37.749207 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.749182    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:37.849754 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:37.849687    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:39.305264 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:39.305224    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:39.360493 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:39.360464    2583 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 01 12:21:39.360911 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:39.360590    2583 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 01 12:21:39.360911 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:39.360663    2583 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://abc0a0f33396342028b8d407fe49f7f2-4917c070ca97fa5c.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-140-168.us-west-2.compute.internal.18e3eb36c4b56879\": read tcp 10.0.140.168:54164->3.136.89.54:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-140-168.us-west-2.compute.internal.18e3eb36c4b56879  default   5355 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-140-168.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-140-168.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-140-168.us-west-2.compute.internal,},FirstTimestamp:2026-11-01 12:21:37 +0000 UTC,LastTimestamp:2026-11-01 12:21:37.383805601 +0000 UTC m=+0.663733148,Count:4,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-140-168.us-west-2.compute.internal,}"
Nov 01 12:21:39.405965 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:39.405923    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:39.506596 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:39.506530    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:39.606679 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:39.606627    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:39.653018 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:39.652957    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 12:16:38 +0000 UTC" deadline="2028-07-03 16:48:40.642926867 +0000 UTC"
Nov 01 12:21:39.653018 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:39.653009    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14644h27m0.98992088s"
Nov 01 12:21:39.674135 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:39.674112    2583 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:21:39.677226 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:39.677210    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:21:39.677305 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:39.677242    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:21:39.677305 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:39.677254    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:21:39.706748 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:39.706717    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:39.807447 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:39.807369    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:39.907945 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:39.907903    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:40.008464 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:40.008430    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:40.109156 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:40.109095    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:40.130207 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.130183    2583 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:40.209240 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:40.209209    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:40.274024 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:40.273989    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ee0d9d2202565b503aec356e54ab8fe.slice/crio-15fe03495c807090f05994076727b74d2a9b93b7082764faaf38c0cd24672c89 WatchSource:0}: Error finding container 15fe03495c807090f05994076727b74d2a9b93b7082764faaf38c0cd24672c89: Status 404 returned error can't find the container with id 15fe03495c807090f05994076727b74d2a9b93b7082764faaf38c0cd24672c89
Nov 01 12:21:40.274366 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:40.274344    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7042c7ca4bb39d530e5b25fb56952b15.slice/crio-8954c7579906fbf4e5536ed5a8948a443b2697ef0bf87ce1fc9dec9ebc9023e3 WatchSource:0}: Error finding container 8954c7579906fbf4e5536ed5a8948a443b2697ef0bf87ce1fc9dec9ebc9023e3: Status 404 returned error can't find the container with id 8954c7579906fbf4e5536ed5a8948a443b2697ef0bf87ce1fc9dec9ebc9023e3
Nov 01 12:21:40.279147 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.279131    2583 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:21:40.309476 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:40.309444    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:40.410240 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:40.410160    2583 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:21:40.415254 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.415229    2583 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:40.415254 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.415245    2583 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:40.415407 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.415362    2583 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:21:40.419299 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.419287    2583 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 01 12:21:40.421444 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.421431    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:40.478396 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.478367    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:21:40.479335 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.479321    2583 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:21:40.533170 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.533145    2583 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:21:40.619941 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.619913    2583 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 01 12:21:40.620100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.619997    2583 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-dd8mk"
Nov 01 12:21:40.643449 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.643401    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"2ee0d9d2202565b503aec356e54ab8fe","Type":"ContainerStarted","Data":"15fe03495c807090f05994076727b74d2a9b93b7082764faaf38c0cd24672c89"}
Nov 01 12:21:40.644344 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:40.644327    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"7042c7ca4bb39d530e5b25fb56952b15","Type":"ContainerStarted","Data":"8954c7579906fbf4e5536ed5a8948a443b2697ef0bf87ce1fc9dec9ebc9023e3"}
Nov 01 12:21:41.399422 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.399392    2583 apiserver.go:52] "Watching apiserver"
Nov 01 12:21:41.545010 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.544324    2583 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 12:21:41.545010 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.544691    2583 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ovn-kubernetes/ovnkube-node-hpzwg","kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal","openshift-multus/multus-86859","openshift-multus/network-metrics-daemon-dlcrd","kube-system/global-pull-secret-syncer-5pg82","kube-system/konnectivity-agent-nsn56","openshift-image-registry/node-ca-ld2ml","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx","openshift-dns/node-resolver-wd6jh","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-vw5xj","openshift-multus/multus-additional-cni-plugins-qtgbr","openshift-network-diagnostics/network-check-target-gnbj7","openshift-network-operator/iptables-alerter-bchgz"]
Nov 01 12:21:41.547951 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.547382    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.549634 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.549615    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:41.549760 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.549685    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:41.551851 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.551822    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86859"
Nov 01 12:21:41.554084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.554060    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:41.554239 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.554136    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:41.556241 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.556221    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:41.556331 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.556285    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:41.559151 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.558727    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:41.560760 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.560743    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.564630 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.563270    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.566027 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.565797    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.568252 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.568233    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.571631 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.571610    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.577284 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.577267    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.597335 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.597027    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 12:21:41.597335 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.597174    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 12:21:41.597335 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.597209    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 12:21:41.597335 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.597309    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zhblh\""
Nov 01 12:21:41.600848 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.600703    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 12:21:41.600848 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.600821    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z9pcl\""
Nov 01 12:21:41.621453 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.621435    2583 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 12:21:41.624849 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.624824    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 12:16:40 +0000 UTC" deadline="2028-04-06 18:45:09.190237829 +0000 UTC"
Nov 01 12:21:41.624849 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.624846    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12534h23m27.565394955s"
Nov 01 12:21:41.635251 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635169    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zm6p9\" (UniqueName: \"kubernetes.io/projected/ed6f24b7-7844-4464-9325-e41004b17167-kube-api-access-zm6p9\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.635251 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635208    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-systemd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.635460 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635432    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.635533 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635472    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-script-lib\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.635533 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635496    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-var-lib-kubelet\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.635622 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635535    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-netns\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.635622 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-netd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.635622 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635611    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-config\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.635770 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635635    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-host-slash\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.635770 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635660    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-kubelet-config\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:41.635770 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635685    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:41.635770 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-sys\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.635770 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635732    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a718c0bb-3ac8-432b-9b69-60e61375994f-konnectivity-ca\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:41.635770 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635755    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-kubelet\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635778    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-iptables-alerter-script\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635810    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-daemon-config\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635833    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635857    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-etc-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635890    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-log-socket\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635936    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-systemd-units\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.635986    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-slash\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636019    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636043    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-host\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636074    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.636100 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636099    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636122    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhbsf\" (UniqueName: \"kubernetes.io/projected/c76d9907-a77b-455f-9437-660fd451b40b-kube-api-access-xhbsf\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636146    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cni-binary-copy\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fck6\" (UniqueName: \"kubernetes.io/projected/d19a543b-1191-4040-b996-15e710bbb1bf-kube-api-access-8fck6\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636198    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysconfig\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636225    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636294    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-tuned\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636326    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636354    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636387    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c76d9907-a77b-455f-9437-660fd451b40b-ovn-node-metrics-cert\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636434    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-hostroot\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636461    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636497    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-modprobe-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636523    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-conf\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636560    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-tmp\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636585    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-cnibin\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.636655 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636615    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cnibin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636645    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636674    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-systemd\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636707    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-binary-copy\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636732    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636780    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/246aad42-5862-4e6e-af04-a7a9847f4ab5-serviceca\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636823    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-dbus\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636849    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636875    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-992n4\" (UniqueName: \"kubernetes.io/projected/267252e3-49bf-4478-90fb-2660c06b2b62-kube-api-access-992n4\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636932    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc7p7\" (UniqueName: \"kubernetes.io/projected/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-kube-api-access-cc7p7\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636957    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-var-lib-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.636996    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-node-log\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637023    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-multus\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637048    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-socket-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637070    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-run\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637127    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l59hh\" (UniqueName: \"kubernetes.io/projected/0b6a342b-6f0b-4701-919e-d46c24861d27-kube-api-access-l59hh\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.637436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637166    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-netns\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637191    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-registration-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637216    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-device-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637240    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-k8s-cni-cncf-io\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637265    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-conf-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637296    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-env-overrides\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637321    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2xl9\" (UniqueName: \"kubernetes.io/projected/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-kube-api-access-d2xl9\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637348    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a718c0bb-3ac8-432b-9b69-60e61375994f-agent-certs\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637372    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-system-cni-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637396    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-system-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637419    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-ovn\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637442    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-bin\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637465    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/246aad42-5862-4e6e-af04-a7a9847f4ab5-host\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637547    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g877r\" (UniqueName: \"kubernetes.io/projected/246aad42-5862-4e6e-af04-a7a9847f4ab5-kube-api-access-g877r\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637579    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-socket-dir-parent\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637618    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-bin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638128 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637649    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-kubelet\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638781 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637688    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-etc-kubernetes\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638781 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637742    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drtsk\" (UniqueName: \"kubernetes.io/projected/04bb07a5-23a4-4095-a019-cb8a36e1bd63-kube-api-access-drtsk\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638781 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.637791    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6f24b7-7844-4464-9325-e41004b17167-tmp-dir\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.638781 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.638095    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-kubernetes\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.638781 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.638121    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-lib-modules\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.638781 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.638144    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-sys-fs\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.638781 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.638161    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-multus-certs\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.638781 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.638175    2583 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed6f24b7-7844-4464-9325-e41004b17167-hosts-file\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.640783 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.640762    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 12:21:41.640953 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.640926    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nsq77\""
Nov 01 12:21:41.641030 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.640991    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 12:21:41.641078 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641064    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7twmh\""
Nov 01 12:21:41.641078 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641067    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641344    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641400    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641445    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641559    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641626    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641650    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641689    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641698    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641894    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 12:21:41.641994 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.641939    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 12:21:41.644661 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.644530    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qrhfv\""
Nov 01 12:21:41.644661 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.644598    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 12:21:41.644661 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.644623    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9cl4q\""
Nov 01 12:21:41.644661 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.644631    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhf7p\""
Nov 01 12:21:41.647288 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.647271    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 12:21:41.688413 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688347    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 12:21:41.688413 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688378    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2z6g\""
Nov 01 12:21:41.688413 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688408    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 12:21:41.688608 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688357    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:21:41.688608 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688419    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 12:21:41.688608 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688430    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 12:21:41.688608 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688452    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 12:21:41.688608 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688490    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 12:21:41.688608 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688500    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 12:21:41.688802 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.688743    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sfjvd\""
Nov 01 12:21:41.738725 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738701    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cni-binary-copy\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.738869 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738737    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fck6\" (UniqueName: \"kubernetes.io/projected/d19a543b-1191-4040-b996-15e710bbb1bf-kube-api-access-8fck6\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.738869 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738762    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysconfig\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.738869 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738788    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.738869 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738827    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-tuned\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.738869 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738851    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.739079 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738881    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:41.739079 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738907    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c76d9907-a77b-455f-9437-660fd451b40b-ovn-node-metrics-cert\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.739079 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738929    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-hostroot\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.739079 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738950    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.739079 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.738990    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-modprobe-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.739079 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739015    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-conf\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.739079 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739036    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-tmp\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.739079 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739074    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-cnibin\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739095    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cnibin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739117    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739142    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-systemd\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739167    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-binary-copy\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739190    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739214    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/246aad42-5862-4e6e-af04-a7a9847f4ab5-serviceca\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739238    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-dbus\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739260    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739285    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992n4\" (UniqueName: \"kubernetes.io/projected/267252e3-49bf-4478-90fb-2660c06b2b62-kube-api-access-992n4\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739308    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc7p7\" (UniqueName: \"kubernetes.io/projected/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-kube-api-access-cc7p7\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739328    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-var-lib-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739343    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-node-log\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739359    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-multus\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739375    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-socket-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739394    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-run\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739408    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59hh\" (UniqueName: \"kubernetes.io/projected/0b6a342b-6f0b-4701-919e-d46c24861d27-kube-api-access-l59hh\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.739433 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739426    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-netns\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739450    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-registration-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739471    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-device-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739485    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-k8s-cni-cncf-io\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739502    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-conf-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739518    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-env-overrides\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739540    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xl9\" (UniqueName: \"kubernetes.io/projected/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-kube-api-access-d2xl9\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739566    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a718c0bb-3ac8-432b-9b69-60e61375994f-agent-certs\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739592    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-system-cni-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739616    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-system-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739641    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-ovn\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739665    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-bin\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739688    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/246aad42-5862-4e6e-af04-a7a9847f4ab5-host\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739711    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g877r\" (UniqueName: \"kubernetes.io/projected/246aad42-5862-4e6e-af04-a7a9847f4ab5-kube-api-access-g877r\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739736    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-socket-dir-parent\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739761    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-bin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739784    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-kubelet\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.740267 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739809    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-etc-kubernetes\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739832    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drtsk\" (UniqueName: \"kubernetes.io/projected/04bb07a5-23a4-4095-a019-cb8a36e1bd63-kube-api-access-drtsk\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739855    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6f24b7-7844-4464-9325-e41004b17167-tmp-dir\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739880    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-kubernetes\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739906    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-lib-modules\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739917    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cni-binary-copy\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739928    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-sys-fs\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739944    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-multus-certs\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.739982    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed6f24b7-7844-4464-9325-e41004b17167-hosts-file\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740010    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm6p9\" (UniqueName: \"kubernetes.io/projected/ed6f24b7-7844-4464-9325-e41004b17167-kube-api-access-zm6p9\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740034    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-systemd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740054    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740072    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-script-lib\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740097    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-var-lib-kubelet\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740123    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-netns\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740139    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-netd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740154    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-config\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741038 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740223    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysconfig\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740261    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-host-slash\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740290    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-kubelet-config\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740320    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740345    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-sys\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740371    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a718c0bb-3ac8-432b-9b69-60e61375994f-konnectivity-ca\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740397    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-kubelet\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740424    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-iptables-alerter-script\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740456    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-daemon-config\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740482    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740510    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-etc-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740539    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-log-socket\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740562    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-config\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740564    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-systemd-units\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740602    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-slash\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740605    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-systemd-units\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740621    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.741723 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740639    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-host\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740655    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740671    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740690    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbsf\" (UniqueName: \"kubernetes.io/projected/c76d9907-a77b-455f-9437-660fd451b40b-kube-api-access-xhbsf\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.740745    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.741033    2583 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.741636    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-hostroot\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.741732    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.741806    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-modprobe-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.741918    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-conf\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.742216    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-kubelet\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.742260    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.742316    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-host-slash\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.742409 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.742377    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-kubelet-config\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:41.743025 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.742488    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:41.743025 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.742559    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret podName:3a7a84c2-0167-4f45-82b8-d8da5f192fac nodeName:}" failed. No retries permitted until 2026-11-01 12:21:42.242538155 +0000 UTC m=+4.167010779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret") pod "global-pull-secret-syncer-5pg82" (UID: "3a7a84c2-0167-4f45-82b8-d8da5f192fac") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:41.743025 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.742630    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-sys\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.743025 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.742850    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-iptables-alerter-script\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.743282 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743230    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a718c0bb-3ac8-432b-9b69-60e61375994f-konnectivity-ca\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:41.743456 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743434    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-daemon-config\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.743584 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.743566    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:41.743643 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743577    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-system-cni-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.743643 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743614    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-netns\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.743643 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.743623    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs podName:752a2f8b-cf54-4bf3-bd4d-39ba739cb627 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:42.243607105 +0000 UTC m=+4.168079707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs") pod "network-metrics-daemon-dlcrd" (UID: "752a2f8b-cf54-4bf3-bd4d-39ba739cb627") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:41.743765 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743653    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-system-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.743765 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743673    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-etc-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.743765 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743687    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-registration-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.743765 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743710    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-ovn\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.743765 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743738    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-device-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.743765 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743752    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-log-socket\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743764    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-bin\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743802    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-k8s-cni-cncf-io\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743815    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/246aad42-5862-4e6e-af04-a7a9847f4ab5-host\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.743866    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-conf-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744329    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-env-overrides\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744395    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-multus\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744412    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-socket-dir-parent\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744438    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-node-log\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744495    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-netd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744558    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-systemd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744529    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-bin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744569    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-slash\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744587    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-kubelet\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744625    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-host\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744634    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744696    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.744680    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-etc-kubernetes\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.745573 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745039    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6f24b7-7844-4464-9325-e41004b17167-tmp-dir\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745134    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-var-lib-kubelet\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745194    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-sys-fs\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745211    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-netns\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745218    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-var-lib-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745311    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-lib-modules\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745385    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-kubernetes\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745444    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-run\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745465    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-multus-certs\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745504    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cnibin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745582    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed6f24b7-7844-4464-9325-e41004b17167-hosts-file\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745678    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-systemd\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745782    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-dbus\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745838    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745914    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.745712    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.746092    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-script-lib\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.746428 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.746360    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a718c0bb-3ac8-432b-9b69-60e61375994f-agent-certs\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:41.747207 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.746484    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-socket-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.747207 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.746534    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-tmp\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.747207 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.746584    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.747207 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.746844    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-binary-copy\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.747207 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.747070    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/246aad42-5862-4e6e-af04-a7a9847f4ab5-serviceca\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.747437 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.747222    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.747437 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.747389    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c76d9907-a77b-455f-9437-660fd451b40b-ovn-node-metrics-cert\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.747600 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.747570    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-cnibin\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.749990 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.748743    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-tuned\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.792452 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.791835    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:41.792452 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.791870    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:41.792452 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.791895    2583 projected.go:196] Error preparing data for projected volume kube-api-access-k467p for pod openshift-network-diagnostics/network-check-target-gnbj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:41.792452 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:41.792010    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p podName:c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:42.291980414 +0000 UTC m=+4.216453033 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-k467p" (UniqueName: "kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p") pod "network-check-target-gnbj7" (UID: "c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:41.794454 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.794365    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fck6\" (UniqueName: \"kubernetes.io/projected/d19a543b-1191-4040-b996-15e710bbb1bf-kube-api-access-8fck6\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.795635 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.795145    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbsf\" (UniqueName: \"kubernetes.io/projected/c76d9907-a77b-455f-9437-660fd451b40b-kube-api-access-xhbsf\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.795635 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.795495    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992n4\" (UniqueName: \"kubernetes.io/projected/267252e3-49bf-4478-90fb-2660c06b2b62-kube-api-access-992n4\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.801899 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.801496    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59hh\" (UniqueName: \"kubernetes.io/projected/0b6a342b-6f0b-4701-919e-d46c24861d27-kube-api-access-l59hh\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.801899 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.801867    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm6p9\" (UniqueName: \"kubernetes.io/projected/ed6f24b7-7844-4464-9325-e41004b17167-kube-api-access-zm6p9\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.803508 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.803484    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g877r\" (UniqueName: \"kubernetes.io/projected/246aad42-5862-4e6e-af04-a7a9847f4ab5-kube-api-access-g877r\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.803606 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.803549    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc7p7\" (UniqueName: \"kubernetes.io/projected/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-kube-api-access-cc7p7\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:41.803956 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.803936    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xl9\" (UniqueName: \"kubernetes.io/projected/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-kube-api-access-d2xl9\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.804513 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.804493    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drtsk\" (UniqueName: \"kubernetes.io/projected/04bb07a5-23a4-4095-a019-cb8a36e1bd63-kube-api-access-drtsk\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:21:41.862170 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.862139    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:21:41.870787 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.870600    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86859"
Nov 01 12:21:41.873629 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:41.873603    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod246aad42_5862_4e6e_af04_a7a9847f4ab5.slice/crio-3a320ab6e458e4d86d522a8be08108efd6e2d46d8681f736235050b789bfd93b WatchSource:0}: Error finding container 3a320ab6e458e4d86d522a8be08108efd6e2d46d8681f736235050b789bfd93b: Status 404 returned error can't find the container with id 3a320ab6e458e4d86d522a8be08108efd6e2d46d8681f736235050b789bfd93b
Nov 01 12:21:41.879396 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:41.879374    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bb07a5_23a4_4095_a019_cb8a36e1bd63.slice/crio-15d3a17c849abcd4ae6b211d7007cc23c4046eadea28261036cbb9139c1ecf02 WatchSource:0}: Error finding container 15d3a17c849abcd4ae6b211d7007cc23c4046eadea28261036cbb9139c1ecf02: Status 404 returned error can't find the container with id 15d3a17c849abcd4ae6b211d7007cc23c4046eadea28261036cbb9139c1ecf02
Nov 01 12:21:41.880155 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.880138    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:41.888876 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.888857    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:21:41.890127 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:41.890108    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda718c0bb_3ac8_432b_9b69_60e61375994f.slice/crio-7ba56b2e4284c5b7d4c3cd52d9ea62847f2a321dd3e1068c26ecce19f95258b3 WatchSource:0}: Error finding container 7ba56b2e4284c5b7d4c3cd52d9ea62847f2a321dd3e1068c26ecce19f95258b3: Status 404 returned error can't find the container with id 7ba56b2e4284c5b7d4c3cd52d9ea62847f2a321dd3e1068c26ecce19f95258b3
Nov 01 12:21:41.896808 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:41.896790    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded6f24b7_7844_4464_9325_e41004b17167.slice/crio-58f62c0c26b1fe16431db461aa138a39f3160518ca3f4db4e24fe7fd8d8d5c2e WatchSource:0}: Error finding container 58f62c0c26b1fe16431db461aa138a39f3160518ca3f4db4e24fe7fd8d8d5c2e: Status 404 returned error can't find the container with id 58f62c0c26b1fe16431db461aa138a39f3160518ca3f4db4e24fe7fd8d8d5c2e
Nov 01 12:21:41.910397 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.910378    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:21:41.915122 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.915104    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:21:41.919081 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.919063    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:21:41.919279 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:41.919256    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5eacd3c_0cc5_4ec9_a161_6d27ba8d3e33.slice/crio-47f48264521834cc2f6b82b67ccc14735ec341f779a26dd48d6bfac50ba29392 WatchSource:0}: Error finding container 47f48264521834cc2f6b82b67ccc14735ec341f779a26dd48d6bfac50ba29392: Status 404 returned error can't find the container with id 47f48264521834cc2f6b82b67ccc14735ec341f779a26dd48d6bfac50ba29392
Nov 01 12:21:41.922336 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.922306    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:21:41.924262 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:41.924242    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc76d9907_a77b_455f_9437_660fd451b40b.slice/crio-e79eaadcd561dd01938bf27ef2210aaeb235fa9fcc6abd9d320ff97ec85ca9d3 WatchSource:0}: Error finding container e79eaadcd561dd01938bf27ef2210aaeb235fa9fcc6abd9d320ff97ec85ca9d3: Status 404 returned error can't find the container with id e79eaadcd561dd01938bf27ef2210aaeb235fa9fcc6abd9d320ff97ec85ca9d3
Nov 01 12:21:41.930147 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:41.930127    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd19a543b_1191_4040_b996_15e710bbb1bf.slice/crio-45eb50a7bf8353c6edbb15bab2fb8c6c7eefb817df10c46c59c819462787f810 WatchSource:0}: Error finding container 45eb50a7bf8353c6edbb15bab2fb8c6c7eefb817df10c46c59c819462787f810: Status 404 returned error can't find the container with id 45eb50a7bf8353c6edbb15bab2fb8c6c7eefb817df10c46c59c819462787f810
Nov 01 12:21:41.933057 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:41.933036    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b6a342b_6f0b_4701_919e_d46c24861d27.slice/crio-f0d2b1e9644567bdc70f31452cb15a30df1b640fef8c88d2667b384b8d65c9e7 WatchSource:0}: Error finding container f0d2b1e9644567bdc70f31452cb15a30df1b640fef8c88d2667b384b8d65c9e7: Status 404 returned error can't find the container with id f0d2b1e9644567bdc70f31452cb15a30df1b640fef8c88d2667b384b8d65c9e7
Nov 01 12:21:41.939252 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:41.939203    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:21:41.949088 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:21:41.949066    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod267252e3_49bf_4478_90fb_2660c06b2b62.slice/crio-63e6d3347c5a8c175c746bc2ee2f0d914839e78aba2355a4bd3fbb7dc93fc25d WatchSource:0}: Error finding container 63e6d3347c5a8c175c746bc2ee2f0d914839e78aba2355a4bd3fbb7dc93fc25d: Status 404 returned error can't find the container with id 63e6d3347c5a8c175c746bc2ee2f0d914839e78aba2355a4bd3fbb7dc93fc25d
Nov 01 12:21:42.243684 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.243611    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:42.243684 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.243653    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:42.243878 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:42.243806    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:42.243878 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:42.243858    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs podName:752a2f8b-cf54-4bf3-bd4d-39ba739cb627 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:43.243841194 +0000 UTC m=+5.168313806 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs") pod "network-metrics-daemon-dlcrd" (UID: "752a2f8b-cf54-4bf3-bd4d-39ba739cb627") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:42.243960 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:42.243915    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:42.243960 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:42.243939    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret podName:3a7a84c2-0167-4f45-82b8-d8da5f192fac nodeName:}" failed. No retries permitted until 2026-11-01 12:21:43.243932169 +0000 UTC m=+5.168404781 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret") pod "global-pull-secret-syncer-5pg82" (UID: "3a7a84c2-0167-4f45-82b8-d8da5f192fac") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:42.344985 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.344932    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:42.345176 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:42.345105    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:42.345176 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:42.345122    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:42.345176 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:42.345134    2583 projected.go:196] Error preparing data for projected volume kube-api-access-k467p for pod openshift-network-diagnostics/network-check-target-gnbj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:42.345176 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:42.345180    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p podName:c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:43.345168294 +0000 UTC m=+5.269640907 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-k467p" (UniqueName: "kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p") pod "network-check-target-gnbj7" (UID: "c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:42.626045 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.625850    2583 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 12:16:40 +0000 UTC" deadline="2028-06-13 20:52:32.901838902 +0000 UTC"
Nov 01 12:21:42.626045 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.626039    2583 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14168h30m50.275803629s"
Nov 01 12:21:42.639768 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.639741    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:42.639910 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:42.639874    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:42.648588 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.648551    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vw5xj" event={"ID":"0b6a342b-6f0b-4701-919e-d46c24861d27","Type":"ContainerStarted","Data":"f0d2b1e9644567bdc70f31452cb15a30df1b640fef8c88d2667b384b8d65c9e7"}
Nov 01 12:21:42.649736 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.649710    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx" event={"ID":"d19a543b-1191-4040-b996-15e710bbb1bf","Type":"ContainerStarted","Data":"45eb50a7bf8353c6edbb15bab2fb8c6c7eefb817df10c46c59c819462787f810"}
Nov 01 12:21:42.650877 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.650857    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd6jh" event={"ID":"ed6f24b7-7844-4464-9325-e41004b17167","Type":"ContainerStarted","Data":"58f62c0c26b1fe16431db461aa138a39f3160518ca3f4db4e24fe7fd8d8d5c2e"}
Nov 01 12:21:42.656515 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.656483    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"2ee0d9d2202565b503aec356e54ab8fe","Type":"ContainerStarted","Data":"2dce942190859e7ef07106b65d9f4b4272fc12997fd95ce9c532ad8a1b1032ef"}
Nov 01 12:21:42.657957 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.657936    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"7042c7ca4bb39d530e5b25fb56952b15","Type":"ContainerStarted","Data":"0a82b62c9f5210a21d4d94805f64e9e63f6a9c2d9f94fed124dc2d49c9d7510d"}
Nov 01 12:21:42.659104 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.659079    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerStarted","Data":"e79eaadcd561dd01938bf27ef2210aaeb235fa9fcc6abd9d320ff97ec85ca9d3"}
Nov 01 12:21:42.660113 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.660092    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bchgz" event={"ID":"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33","Type":"ContainerStarted","Data":"47f48264521834cc2f6b82b67ccc14735ec341f779a26dd48d6bfac50ba29392"}
Nov 01 12:21:42.660988 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.660935    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nsn56" event={"ID":"a718c0bb-3ac8-432b-9b69-60e61375994f","Type":"ContainerStarted","Data":"7ba56b2e4284c5b7d4c3cd52d9ea62847f2a321dd3e1068c26ecce19f95258b3"}
Nov 01 12:21:42.662125 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.662097    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86859" event={"ID":"04bb07a5-23a4-4095-a019-cb8a36e1bd63","Type":"ContainerStarted","Data":"15d3a17c849abcd4ae6b211d7007cc23c4046eadea28261036cbb9139c1ecf02"}
Nov 01 12:21:42.663090 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.663072    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ld2ml" event={"ID":"246aad42-5862-4e6e-af04-a7a9847f4ab5","Type":"ContainerStarted","Data":"3a320ab6e458e4d86d522a8be08108efd6e2d46d8681f736235050b789bfd93b"}
Nov 01 12:21:42.664065 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:42.664040    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtgbr" event={"ID":"267252e3-49bf-4478-90fb-2660c06b2b62","Type":"ContainerStarted","Data":"63e6d3347c5a8c175c746bc2ee2f0d914839e78aba2355a4bd3fbb7dc93fc25d"}
Nov 01 12:21:43.251001 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:43.250909    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:43.251001 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:43.250957    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:43.251219 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.251131    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:43.251219 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.251185    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs podName:752a2f8b-cf54-4bf3-bd4d-39ba739cb627 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:45.251168444 +0000 UTC m=+7.175641052 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs") pod "network-metrics-daemon-dlcrd" (UID: "752a2f8b-cf54-4bf3-bd4d-39ba739cb627") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:43.251343 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.251247    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:43.251343 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.251280    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret podName:3a7a84c2-0167-4f45-82b8-d8da5f192fac nodeName:}" failed. No retries permitted until 2026-11-01 12:21:45.251267761 +0000 UTC m=+7.175740364 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret") pod "global-pull-secret-syncer-5pg82" (UID: "3a7a84c2-0167-4f45-82b8-d8da5f192fac") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:43.292915 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:43.292826    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal" podStartSLOduration=3.292806823 podStartE2EDuration="3.292806823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:43.185627704 +0000 UTC m=+5.110100327" watchObservedRunningTime="2026-11-01 12:21:43.292806823 +0000 UTC m=+5.217279449"
Nov 01 12:21:43.351843 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:43.351806    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:43.352099 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.352011    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:43.352099 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.352031    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:43.352099 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.352044    2583 projected.go:196] Error preparing data for projected volume kube-api-access-k467p for pod openshift-network-diagnostics/network-check-target-gnbj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:43.352279 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.352103    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p podName:c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:45.352085417 +0000 UTC m=+7.276558021 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-k467p" (UniqueName: "kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p") pod "network-check-target-gnbj7" (UID: "c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:43.639584 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:43.639270    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:43.639584 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.639375    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:43.639584 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:43.639443    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:43.639584 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:43.639508    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:43.674426 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:43.674359    2583 generic.go:356] "Generic (PLEG): container finished" podID="2ee0d9d2202565b503aec356e54ab8fe" containerID="2dce942190859e7ef07106b65d9f4b4272fc12997fd95ce9c532ad8a1b1032ef" exitCode=0
Nov 01 12:21:43.675168 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:43.674927    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"2ee0d9d2202565b503aec356e54ab8fe","Type":"ContainerDied","Data":"2dce942190859e7ef07106b65d9f4b4272fc12997fd95ce9c532ad8a1b1032ef"}
Nov 01 12:21:44.640793 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:44.640748    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:44.641294 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:44.640881    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:44.680535 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:44.680500    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"2ee0d9d2202565b503aec356e54ab8fe","Type":"ContainerStarted","Data":"9822ffca28abb0c2a170ebc5332d71ef2d8b393298241ff8bb177baa2f774661"}
Nov 01 12:21:44.787636 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:44.787582    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal" podStartSLOduration=4.787562453 podStartE2EDuration="4.787562453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:44.786759432 +0000 UTC m=+6.711232054" watchObservedRunningTime="2026-11-01 12:21:44.787562453 +0000 UTC m=+6.712035076"
Nov 01 12:21:45.269540 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:45.269511    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:45.269661 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:45.269568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:45.269720 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.269710    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:45.269798 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.269777    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs podName:752a2f8b-cf54-4bf3-bd4d-39ba739cb627 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:49.269757759 +0000 UTC m=+11.194230387 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs") pod "network-metrics-daemon-dlcrd" (UID: "752a2f8b-cf54-4bf3-bd4d-39ba739cb627") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:45.269867 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.269840    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:45.269913 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.269875    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret podName:3a7a84c2-0167-4f45-82b8-d8da5f192fac nodeName:}" failed. No retries permitted until 2026-11-01 12:21:49.269864599 +0000 UTC m=+11.194337212 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret") pod "global-pull-secret-syncer-5pg82" (UID: "3a7a84c2-0167-4f45-82b8-d8da5f192fac") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:45.370613 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:45.370568    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:45.370884 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.370782    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:45.370884 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.370805    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:45.370884 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.370817    2583 projected.go:196] Error preparing data for projected volume kube-api-access-k467p for pod openshift-network-diagnostics/network-check-target-gnbj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:45.370884 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.370872    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p podName:c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:49.370854805 +0000 UTC m=+11.295327408 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-k467p" (UniqueName: "kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p") pod "network-check-target-gnbj7" (UID: "c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:45.638781 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:45.638700    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:45.639024 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.638830    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:45.639243 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:45.639226    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:45.639345 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:45.639328    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:45.691349 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:45.691315    2583 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="70b5ba8c9fa4487aaaeb93c1da024995dc2732d44cd6f39627ca36ee24f002bc" exitCode=0
Nov 01 12:21:45.692534 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:45.692486    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtgbr" event={"ID":"267252e3-49bf-4478-90fb-2660c06b2b62","Type":"ContainerDied","Data":"70b5ba8c9fa4487aaaeb93c1da024995dc2732d44cd6f39627ca36ee24f002bc"}
Nov 01 12:21:46.639436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:46.638966    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:46.639436 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:46.639096    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:46.698436 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:46.697709    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-nsn56" event={"ID":"a718c0bb-3ac8-432b-9b69-60e61375994f","Type":"ContainerStarted","Data":"7f649d0fe2396744b80cebc1dcc26c618d9aa38a8ce931743e22f0e6a9a8089a"}
Nov 01 12:21:46.805359 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:46.805173    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-nsn56" podStartSLOduration=4.672271964 podStartE2EDuration="8.805155293s" totalImagesPullingTime="4.132883329s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:46.804060104 +0000 UTC m=+8.728532725" watchObservedRunningTime="2026-11-01 12:21:46.805155293 +0000 UTC m=+8.729627919"
Nov 01 12:21:47.639028 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:47.638878    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:47.639884 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:47.639296    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:47.639884 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:47.639680    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:47.639884 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:47.639790    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:47.703347 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:47.703249    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ld2ml" event={"ID":"246aad42-5862-4e6e-af04-a7a9847f4ab5","Type":"ContainerStarted","Data":"fdeedee5678c33af82b2cf60b6651d9752c5d94c29e44537df33e8b0c3c31e6f"}
Nov 01 12:21:47.808988 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:47.808865    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ld2ml" podStartSLOduration=4.443762922 podStartE2EDuration="9.80884542s" totalImagesPullingTime="5.365082498s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:47.808523145 +0000 UTC m=+9.732995768" watchObservedRunningTime="2026-11-01 12:21:47.80884542 +0000 UTC m=+9.733318043"
Nov 01 12:21:48.649833 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:48.649798    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:48.650181 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:48.650148    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:48.708295 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:48.708263    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx" event={"ID":"d19a543b-1191-4040-b996-15e710bbb1bf","Type":"ContainerStarted","Data":"03d54885a9afe86cdce89d735c942395124f70733bf268bf4d233cb2241fb86b"}
Nov 01 12:21:49.306245 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:49.306192    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:49.306245 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:49.306240    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:49.306474 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.306403    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:49.306474 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.306462    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs podName:752a2f8b-cf54-4bf3-bd4d-39ba739cb627 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:57.306447678 +0000 UTC m=+19.230920291 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs") pod "network-metrics-daemon-dlcrd" (UID: "752a2f8b-cf54-4bf3-bd4d-39ba739cb627") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:49.306874 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.306835    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:49.306874 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.306874    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret podName:3a7a84c2-0167-4f45-82b8-d8da5f192fac nodeName:}" failed. No retries permitted until 2026-11-01 12:21:57.30686418 +0000 UTC m=+19.231336793 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret") pod "global-pull-secret-syncer-5pg82" (UID: "3a7a84c2-0167-4f45-82b8-d8da5f192fac") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:49.407148 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:49.407108    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:49.407320 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.407294    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:49.407320 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.407315    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:49.407422 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.407329    2583 projected.go:196] Error preparing data for projected volume kube-api-access-k467p for pod openshift-network-diagnostics/network-check-target-gnbj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:49.407422 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.407392    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p podName:c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866 nodeName:}" failed. No retries permitted until 2026-11-01 12:21:57.407374022 +0000 UTC m=+19.331846639 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-k467p" (UniqueName: "kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p") pod "network-check-target-gnbj7" (UID: "c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:49.638633 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:49.638345    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:49.638633 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.638484    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:49.638633 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:49.638550    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:49.638633 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:49.638606    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:49.713525 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:49.713481    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86859" event={"ID":"04bb07a5-23a4-4095-a019-cb8a36e1bd63","Type":"ContainerStarted","Data":"a13eb08c5eb73819bf64648b47e0e539eafbfedff47a24539a8b335a5391bf18"}
Nov 01 12:21:50.172116 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.171984    2583 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 12:21:50.631946 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.631582    2583 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T12:21:50.172007226Z","UUID":"9c36b033-6f4f-4a4e-9346-8c6342d8a756","Handler":null,"Name":"","Endpoint":""}
Nov 01 12:21:50.633402 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.633376    2583 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 12:21:50.633518 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.633410    2583 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 12:21:50.638930 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.638756    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:50.638930 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:50.638882    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:50.718073 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.717995    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx" event={"ID":"d19a543b-1191-4040-b996-15e710bbb1bf","Type":"ContainerStarted","Data":"bea2775420fa51b17c628b4640111220996c0a27e52cc45088e021adfc135e50"}
Nov 01 12:21:50.719609 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.719571    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wd6jh" event={"ID":"ed6f24b7-7844-4464-9325-e41004b17167","Type":"ContainerStarted","Data":"c7814c9f7a6a6cfecb8283ba1db5c35f1b87e107f63864b6119d0d2ec4f6c640"}
Nov 01 12:21:50.721735 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.721708    2583 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="1413e38a298ea2b09f78dd20eb139c25fdd1c2070e63b40a5e992edf050a1051" exitCode=0
Nov 01 12:21:50.721842 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.721803    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtgbr" event={"ID":"267252e3-49bf-4478-90fb-2660c06b2b62","Type":"ContainerDied","Data":"1413e38a298ea2b09f78dd20eb139c25fdd1c2070e63b40a5e992edf050a1051"}
Nov 01 12:21:50.826119 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.826062    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-86859" podStartSLOduration=5.163413129 podStartE2EDuration="12.826045521s" totalImagesPullingTime="7.662632392s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:49.827153219 +0000 UTC m=+11.751625842" watchObservedRunningTime="2026-11-01 12:21:50.826045521 +0000 UTC m=+12.750518143"
Nov 01 12:21:50.826513 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.826484    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-wd6jh" podStartSLOduration=4.425453834 podStartE2EDuration="12.82647788s" totalImagesPullingTime="8.401024046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:50.825894538 +0000 UTC m=+12.750367161" watchObservedRunningTime="2026-11-01 12:21:50.82647788 +0000 UTC m=+12.750950501"
Nov 01 12:21:50.892354 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.892309    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:50.892800 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:50.892777    2583 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:21:51.638480 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:51.638388    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:51.638480 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:51.638437    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:51.638729 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:51.638556    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:51.639014 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:51.638946    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:51.726492 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:51.726447    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bchgz" event={"ID":"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33","Type":"ContainerStarted","Data":"fba55d42528f63d86fca02a009dd4d43da93cce07e7a99c827a85565f2be155e"}
Nov 01 12:21:52.639315 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:52.639222    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:52.639460 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:52.639356    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:52.730951 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:52.730916    2583 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="c75c6aae2760ca55ec2675f7a9b917119eed2652c553b3dd4dd6b238d4457654" exitCode=0
Nov 01 12:21:52.731620 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:52.731006    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtgbr" event={"ID":"267252e3-49bf-4478-90fb-2660c06b2b62","Type":"ContainerDied","Data":"c75c6aae2760ca55ec2675f7a9b917119eed2652c553b3dd4dd6b238d4457654"}
Nov 01 12:21:52.733233 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:52.733212    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-vw5xj" event={"ID":"0b6a342b-6f0b-4701-919e-d46c24861d27","Type":"ContainerStarted","Data":"3a39f4f0be9249f16a4f99139c555fa818311bd59f5b2d495d034fa2a2710c9c"}
Nov 01 12:21:52.735537 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:52.735506    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx" event={"ID":"d19a543b-1191-4040-b996-15e710bbb1bf","Type":"ContainerStarted","Data":"72337be306e3d88acc96644aaf6b6e3b0268f8e119da97dbfbcdd95dc31f1ce9"}
Nov 01 12:21:52.936660 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:52.936570    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bchgz" podStartSLOduration=6.946336179 podStartE2EDuration="14.936557071s" totalImagesPullingTime="7.990220892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:51.832817469 +0000 UTC m=+13.757290090" watchObservedRunningTime="2026-11-01 12:21:52.936557071 +0000 UTC m=+14.861029691"
Nov 01 12:21:53.044083 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:53.044031    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-vw5xj" podStartSLOduration=4.837770093 podStartE2EDuration="15.044010253s" totalImagesPullingTime="10.20624016s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:53.043868812 +0000 UTC m=+14.968341434" watchObservedRunningTime="2026-11-01 12:21:53.044010253 +0000 UTC m=+14.968482874"
Nov 01 12:21:53.153030 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:53.152951    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx" podStartSLOduration=5.666343327 podStartE2EDuration="15.152933022s" totalImagesPullingTime="9.486589695s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:53.151859276 +0000 UTC m=+15.076331896" watchObservedRunningTime="2026-11-01 12:21:53.152933022 +0000 UTC m=+15.077405644"
Nov 01 12:21:53.639143 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:53.639105    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:53.639301 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:53.639119    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:53.639301 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:53.639239    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:53.639301 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:53.639277    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:54.639325 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:54.639034    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:54.639891 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:54.639364    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:54.743117 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:54.743078    2583 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="0bb59d2f16b886fa9cc2dbc520ed260bf61fb763f61bfc3833fb5b16dbef2810" exitCode=0
Nov 01 12:21:54.743259 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:54.743133    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtgbr" event={"ID":"267252e3-49bf-4478-90fb-2660c06b2b62","Type":"ContainerDied","Data":"0bb59d2f16b886fa9cc2dbc520ed260bf61fb763f61bfc3833fb5b16dbef2810"}
Nov 01 12:21:55.638459 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:55.638425    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:55.638639 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:55.638463    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:55.638639 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:55.638559    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:55.638736 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:55.638682    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:56.639431 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:56.639402    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:56.639750 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:56.639536    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:56.751754 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:56.751731    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:21:56.752074 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:56.752054    2583 generic.go:356] "Generic (PLEG): container finished" podID="c76d9907-a77b-455f-9437-660fd451b40b" containerID="8f0d60fb59addc8e6846e68ad44b5b7a37849f102611e10125629173a067dfb2" exitCode=1
Nov 01 12:21:56.752123 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:56.752108    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerStarted","Data":"9409d1506180969ad9dea0e31ec5c29ceed8aab9f870b8b01a8e4910251c3e85"}
Nov 01 12:21:56.752169 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:56.752159    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerStarted","Data":"4ab77a5fa23c923c4b90f6c0a968d6170aa82e51380856cf3eae89d053efa7d2"}
Nov 01 12:21:56.752211 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:56.752191    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerStarted","Data":"d4ceaf10fce7d6c4527e239ed1d12d6c8db4b5da8eb810b7a9a69a99a4fda084"}
Nov 01 12:21:56.752242 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:56.752220    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerDied","Data":"8f0d60fb59addc8e6846e68ad44b5b7a37849f102611e10125629173a067dfb2"}
Nov 01 12:21:56.752273 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:56.752247    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerStarted","Data":"05491e47e637ad82b974131fa1f2eecbe33ab70ca076bbe59cb49afadfadddc4"}
Nov 01 12:21:57.372028 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:57.371830    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:57.372028 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:57.372035    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:57.372230 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.371964    2583 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:57.372230 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.372116    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:57.372230 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.372134    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret podName:3a7a84c2-0167-4f45-82b8-d8da5f192fac nodeName:}" failed. No retries permitted until 2026-11-01 12:22:13.372119553 +0000 UTC m=+35.296592165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret") pod "global-pull-secret-syncer-5pg82" (UID: "3a7a84c2-0167-4f45-82b8-d8da5f192fac") : object "kube-system"/"original-pull-secret" not registered
Nov 01 12:21:57.372230 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.372151    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs podName:752a2f8b-cf54-4bf3-bd4d-39ba739cb627 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:13.372140507 +0000 UTC m=+35.296613122 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs") pod "network-metrics-daemon-dlcrd" (UID: "752a2f8b-cf54-4bf3-bd4d-39ba739cb627") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 01 12:21:57.472279 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:57.472251    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:57.472420 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.472358    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 01 12:21:57.472420 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.472370    2583 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 01 12:21:57.472420 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.472378    2583 projected.go:196] Error preparing data for projected volume kube-api-access-k467p for pod openshift-network-diagnostics/network-check-target-gnbj7: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:57.472420 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.472417    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p podName:c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:13.472406232 +0000 UTC m=+35.396878832 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-k467p" (UniqueName: "kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p") pod "network-check-target-gnbj7" (UID: "c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 01 12:21:57.639261 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:57.639198    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:57.639378 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.639297    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:57.639378 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:57.639352    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:57.639461 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:57.639444    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:57.756508 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:57.756481    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:21:57.756896 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:57.756872    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerStarted","Data":"c7fd71de0a46b0b63831619e35231f7ad179e65b618ea022d82bafab48b7fed5"}
Nov 01 12:21:57.759353 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:57.759330    2583 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="abe54d15ad0852976f83f40b1b7f3db3c49943a3524eddf203dcd86f179c314f" exitCode=0
Nov 01 12:21:57.759477 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:57.759364    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtgbr" event={"ID":"267252e3-49bf-4478-90fb-2660c06b2b62","Type":"ContainerDied","Data":"abe54d15ad0852976f83f40b1b7f3db3c49943a3524eddf203dcd86f179c314f"}
Nov 01 12:21:58.639237 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:58.639203    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:21:58.639402 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:58.639281    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:21:58.764217 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:58.764178    2583 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="4b2851e53148e87e11af6752f3e565eb4884c387376dd3c36510373abd4a5712" exitCode=0
Nov 01 12:21:58.764710 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:58.764251    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtgbr" event={"ID":"267252e3-49bf-4478-90fb-2660c06b2b62","Type":"ContainerDied","Data":"4b2851e53148e87e11af6752f3e565eb4884c387376dd3c36510373abd4a5712"}
Nov 01 12:21:59.639245 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:59.639210    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:21:59.639424 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:59.639387    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:21:59.639514 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:59.639426    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:21:59.639514 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:21:59.639491    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:21:59.769612 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:59.769585    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:21:59.770020 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:59.769991    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerStarted","Data":"d93fa2f94caf48d8332b5169386ec0771c84ebbd925bb890a2b439325d167a31"}
Nov 01 12:21:59.772827 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:59.772797    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-qtgbr" event={"ID":"267252e3-49bf-4478-90fb-2660c06b2b62","Type":"ContainerStarted","Data":"f21dbb72edb54ee2a758dedcb302a9f230805e228191a1e4ca9095aad4c57864"}
Nov 01 12:21:59.888112 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:21:59.888062    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-qtgbr" podStartSLOduration=9.346094553 podStartE2EDuration="21.888050137s" totalImagesPullingTime="12.541955584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:21:59.887876104 +0000 UTC m=+21.812348725" watchObservedRunningTime="2026-11-01 12:21:59.888050137 +0000 UTC m=+21.812522758"
Nov 01 12:22:00.638963 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:00.638925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:00.639136 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:00.639066    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:22:01.639222 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:01.639187    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:01.639790 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:01.639303    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:22:01.639790 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:01.639354    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:01.639790 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:01.639434    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:22:01.780135 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:01.780111    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:22:01.780483 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:01.780464    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerStarted","Data":"0e2e8b6eb65147fe48e01a14da70cc714e487cee009f105a75ab23604de950ee"}
Nov 01 12:22:01.780749 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:01.780731    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:01.780835 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:01.780813    2583 scope.go:122] "RemoveContainer" containerID="8f0d60fb59addc8e6846e68ad44b5b7a37849f102611e10125629173a067dfb2"
Nov 01 12:22:01.796227 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:01.796210    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:02.638671 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:02.638641    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:02.638889 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:02.638750    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:22:02.786426 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:02.786399    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:22:02.786816 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:02.786708    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" event={"ID":"c76d9907-a77b-455f-9437-660fd451b40b","Type":"ContainerStarted","Data":"3c729b94a6048b2d4d1f2c6c243afefd61c21b485f2c5e8212765d30b8c91d20"}
Nov 01 12:22:02.786816 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:02.786806    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 12:22:02.787101 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:02.787086    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:02.802304 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:02.802285    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:02.979469 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:02.979378    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg" podStartSLOduration=10.652505974 podStartE2EDuration="24.979364203s" totalImagesPullingTime="14.326858229s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:02.948701235 +0000 UTC m=+24.873173856" watchObservedRunningTime="2026-11-01 12:22:02.979364203 +0000 UTC m=+24.903836824"
Nov 01 12:22:03.638567 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:03.638486    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:03.638567 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:03.638502    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:03.638787 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:03.638611    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:22:03.638787 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:03.638686    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:22:03.649683 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:03.649650    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5pg82"]
Nov 01 12:22:03.654703 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:03.654675    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gnbj7"]
Nov 01 12:22:03.654936 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:03.654913    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dlcrd"]
Nov 01 12:22:03.655073 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:03.655049    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:03.655199 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:03.655176    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:22:03.788578 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:03.788545    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:03.788578 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:03.788568    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:03.789082 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:03.788678    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:22:03.789082 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:03.788696    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 12:22:03.789082 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:03.788769    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:22:04.790533 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:04.790508    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 12:22:05.638768 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:05.638737    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:05.638768 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:05.638754    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:05.638985 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:05.638783    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:05.638985 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:05.638915    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:22:05.639087 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:05.639036    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:22:05.639131 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:05.639098    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:22:07.638487 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:07.638455    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:07.638874 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:07.638490    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:07.638874 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:07.638464    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:07.638874 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:07.638555    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gnbj7" podUID="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866"
Nov 01 12:22:07.638874 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:07.638645    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dlcrd" podUID="752a2f8b-cf54-4bf3-bd4d-39ba739cb627"
Nov 01 12:22:07.638874 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:07.638705    2583 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5pg82" podUID="3a7a84c2-0167-4f45-82b8-d8da5f192fac"
Nov 01 12:22:07.848714 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:07.848676    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:07.848913 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:07.848900    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 12:22:07.863635 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:07.863580    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:08.426720 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:08.426654    2583 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeReady"
Nov 01 12:22:08.426848 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:08.426775    2583 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 01 12:22:09.584006 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.583803    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:22:09.584400 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.584094    2583 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 12:22:09.584435 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.584399    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:22:09.638709 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.638683    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:09.638837 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.638804    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:09.638952 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.638925    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:09.722108 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.722067    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 12:22:09.729448 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.729414    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 12:22:09.729612 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.729517    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-phch9\""
Nov 01 12:22:09.789343 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.789314    2583 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 12:22:09.791744 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.791729    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9t2fc\""
Nov 01 12:22:09.791819 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.791746    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 12:22:09.791857 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:09.791841    2583 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzkjs\""
Nov 01 12:22:13.386116 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:13.386080    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:13.386116 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:13.386119    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:13.386717 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:13.386312    2583 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 01 12:22:13.386717 ip-10-0-140-168 kubenswrapper[2583]: E1101 12:22:13.386413    2583 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs podName:752a2f8b-cf54-4bf3-bd4d-39ba739cb627 nodeName:}" failed. No retries permitted until 2026-11-01 12:22:45.386387728 +0000 UTC m=+67.310860341 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs") pod "network-metrics-daemon-dlcrd" (UID: "752a2f8b-cf54-4bf3-bd4d-39ba739cb627") : secret "metrics-daemon-secret" not found
Nov 01 12:22:13.389960 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:13.389938    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:13.487084 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:13.487016    2583 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:13.489557 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:13.489528    2583 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:13.578262 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:13.578235    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:13.596553 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:13.596529    2583 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:13.883920 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:13.883895    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wd6jh_ed6f24b7-7844-4464-9325-e41004b17167/dns-node-resolver/0.log"
Nov 01 12:22:14.007469 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:14.007431    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5pg82"]
Nov 01 12:22:14.010382 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:22:14.010360    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7a84c2_0167_4f45_82b8_d8da5f192fac.slice/crio-0452ea431d5404e55d2961d22e965674db16147e36a067c7089e4d401140c341 WatchSource:0}: Error finding container 0452ea431d5404e55d2961d22e965674db16147e36a067c7089e4d401140c341: Status 404 returned error can't find the container with id 0452ea431d5404e55d2961d22e965674db16147e36a067c7089e4d401140c341
Nov 01 12:22:14.053624 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:14.053605    2583 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gnbj7"]
Nov 01 12:22:14.056344 ip-10-0-140-168 kubenswrapper[2583]: W1101 12:22:14.056318    2583 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc13b7ab6_ee4e_4128_8e8a_1f4d7fe3c866.slice/crio-1cf85aebf6b03fffe41cbb895bdca1535405fcaa66abc6353ef79fcef60a4515 WatchSource:0}: Error finding container 1cf85aebf6b03fffe41cbb895bdca1535405fcaa66abc6353ef79fcef60a4515: Status 404 returned error can't find the container with id 1cf85aebf6b03fffe41cbb895bdca1535405fcaa66abc6353ef79fcef60a4515
Nov 01 12:22:14.732679 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:14.732651    2583 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ld2ml_246aad42-5862-4e6e-af04-a7a9847f4ab5/node-ca/0.log"
Nov 01 12:22:14.814228 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:14.814192    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5pg82" event={"ID":"3a7a84c2-0167-4f45-82b8-d8da5f192fac","Type":"ContainerStarted","Data":"0452ea431d5404e55d2961d22e965674db16147e36a067c7089e4d401140c341"}
Nov 01 12:22:14.815243 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:14.815220    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gnbj7" event={"ID":"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866","Type":"ContainerStarted","Data":"1cf85aebf6b03fffe41cbb895bdca1535405fcaa66abc6353ef79fcef60a4515"}
Nov 01 12:22:18.827476 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:18.827200    2583 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gnbj7" event={"ID":"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866","Type":"ContainerStarted","Data":"59b2e4468652dcdb53474a5afdbb3d219383b160f4854413bb1f3543a473429a"}
Nov 01 12:22:18.827476 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:18.827469    2583 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:18.934729 ip-10-0-140-168 kubenswrapper[2583]: I1101 12:22:18.934663    2583 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gnbj7" podStartSLOduration=36.501666037 podStartE2EDuration="40.934647321s" totalImagesPullingTime="4.432981284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:18.934230485 +0000 UTC m=+40.858703107" watchObservedRunningTime="2026-11-01 12:22:18.934647321 +0000 UTC m=+40.859119941"
Nov 01 12:22:20.026704 ip-10-0-140-168 systemd[1]: Stopping Kubernetes Kubelet...
Nov 01 12:22:20.041456 ip-10-0-140-168 systemd[1]: kubelet.service: Deactivated successfully.
Nov 01 12:22:20.041672 ip-10-0-140-168 systemd[1]: Stopped Kubernetes Kubelet.
Nov 01 12:22:20.041870 ip-10-0-140-168 systemd[1]: kubelet.service: Consumed 2.216s CPU time.
Nov 01 12:22:20.051561 ip-10-0-140-168 systemd[1]: Starting Kubernetes Kubelet...
Nov 01 12:22:20.112288 ip-10-0-140-168 kubenswrapper[4095]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:22:20.112288 ip-10-0-140-168 kubenswrapper[4095]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 01 12:22:20.112288 ip-10-0-140-168 kubenswrapper[4095]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:22:20.112288 ip-10-0-140-168 kubenswrapper[4095]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 01 12:22:20.114377 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114306    4095 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:20.114377 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114376    4095 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:20.114377 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114380    4095 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114383    4095 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114386    4095 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114389    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114392    4095 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114394    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114397    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114400    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114402    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114405    4095 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114407    4095 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114410    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114412    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114415    4095 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114419    4095 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114421    4095 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114424    4095 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114426    4095 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114429    4095 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114431    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:20.114476 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114434    4095 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114437    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114443    4095 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114446    4095 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114449    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114451    4095 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114454    4095 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114456    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114459    4095 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114461    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114464    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114467    4095 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114469    4095 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114472    4095 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114475    4095 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114478    4095 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114481    4095 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114483    4095 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114486    4095 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114489    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:20.115017 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114492    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114494    4095 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114497    4095 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114500    4095 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114502    4095 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114505    4095 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114507    4095 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114510    4095 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114512    4095 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114515    4095 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114517    4095 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114520    4095 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114522    4095 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114525    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114528    4095 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114531    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114533    4095 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114536    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114538    4095 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:20.115517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114541    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114543    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114547    4095 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114550    4095 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114552    4095 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114555    4095 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114558    4095 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114560    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114563    4095 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114565    4095 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114568    4095 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114571    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114573    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114577    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114586    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114589    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114592    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114594    4095 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114597    4095 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114600    4095 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:20.116015 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114603    4095 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114605    4095 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114608    4095 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114610    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114613    4095 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114615    4095 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114618    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114621    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114624    4095 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114627    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114629    4095 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114632    4095 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114634    4095 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114637    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114640    4095 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114643    4095 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114645    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114648    4095 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114653    4095 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:20.116508 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114657    4095 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114660    4095 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114663    4095 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114665    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114668    4095 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.114673    4095 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115434    4095 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115441    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115444    4095 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115446    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115449    4095 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115452    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115455    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115457    4095 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115460    4095 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115462    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115465    4095 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115467    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115470    4095 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:20.116962 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115472    4095 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115475    4095 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115477    4095 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115481    4095 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115483    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115486    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115488    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115492    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115495    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115498    4095 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115500    4095 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115504    4095 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115508    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115511    4095 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115514    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115519    4095 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115523    4095 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115526    4095 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115530    4095 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:20.117434 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115533    4095 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115536    4095 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115539    4095 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115541    4095 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115544    4095 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115547    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115549    4095 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115552    4095 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115555    4095 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115558    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115560    4095 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115563    4095 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115566    4095 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115568    4095 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115571    4095 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115574    4095 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115576    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115579    4095 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115582    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115585    4095 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:20.117908 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115587    4095 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115590    4095 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115593    4095 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115595    4095 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115599    4095 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115601    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115604    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115607    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115609    4095 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115612    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115614    4095 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115617    4095 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115619    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115622    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115624    4095 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115627    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115629    4095 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115632    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115634    4095 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:20.118425 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115637    4095 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115640    4095 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115642    4095 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115645    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115648    4095 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115650    4095 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115653    4095 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115659    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115661    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115664    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115667    4095 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115670    4095 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115672    4095 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115675    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115678    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115680    4095 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115683    4095 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115686    4095 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115688    4095 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:20.118917 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115691    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115694    4095 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115696    4095 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115699    4095 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115701    4095 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115704    4095 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115706    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115709    4095 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115711    4095 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115714    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115716    4095 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115720    4095 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115722    4095 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115725    4095 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115728    4095 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.115730    4095 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116198    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116203    4095 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116206    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116209    4095 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:20.119461 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116214    4095 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116217    4095 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116220    4095 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116222    4095 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116225    4095 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116228    4095 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116231    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116233    4095 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116236    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116238    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116241    4095 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116243    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116246    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116249    4095 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116251    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116254    4095 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116256    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116259    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116261    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116264    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:20.119948 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116267    4095 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116269    4095 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116272    4095 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116276    4095 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116279    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116282    4095 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116285    4095 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116287    4095 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116290    4095 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116292    4095 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116294    4095 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116297    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116301    4095 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116304    4095 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116307    4095 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116310    4095 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116312    4095 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116315    4095 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116320    4095 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:20.120449 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116324    4095 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116327    4095 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116330    4095 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116333    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116335    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116338    4095 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116340    4095 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116343    4095 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116345    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116348    4095 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116350    4095 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116353    4095 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116356    4095 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116358    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116361    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116363    4095 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116366    4095 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116368    4095 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116371    4095 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:20.120916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116374    4095 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116376    4095 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116378    4095 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116381    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116383    4095 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116386    4095 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116390    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116393    4095 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116396    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116398    4095 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116401    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116403    4095 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116406    4095 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116409    4095 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116411    4095 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116414    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116417    4095 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116419    4095 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116422    4095 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116424    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:20.121623 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116427    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116431    4095 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116433    4095 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116435    4095 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116438    4095 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116441    4095 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116443    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116446    4095 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116448    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116451    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116454    4095 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116456    4095 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116459    4095 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116461    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116464    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116467    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116469    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116471    4095 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116475    4095 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116478    4095 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:20.122399 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116481    4095 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116483    4095 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116487    4095 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.116490    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116576    4095 flags.go:64] FLAG: --address="0.0.0.0"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116587    4095 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116593    4095 flags.go:64] FLAG: --anonymous-auth="true"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116598    4095 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116602    4095 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116605    4095 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116609    4095 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116613    4095 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116616    4095 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116619    4095 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116623    4095 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116626    4095 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116630    4095 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116633    4095 flags.go:64] FLAG: --cgroup-root=""
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116636    4095 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116639    4095 flags.go:64] FLAG: --client-ca-file=""
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116642    4095 flags.go:64] FLAG: --cloud-provider="external"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116645    4095 flags.go:64] FLAG: --cluster-dns="[]"
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116648    4095 flags.go:64] FLAG: --cluster-domain=""
Nov 01 12:22:20.122914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116651    4095 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116654    4095 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116657    4095 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116661    4095 flags.go:64] FLAG: --container-log-max-files="5"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116664    4095 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116667    4095 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116670    4095 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116675    4095 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116678    4095 flags.go:64] FLAG: --contention-profiling="false"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116681    4095 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116685    4095 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116688    4095 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116691    4095 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116695    4095 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116698    4095 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116702    4095 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116705    4095 flags.go:64] FLAG: --enable-load-reader="false"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116707    4095 flags.go:64] FLAG: --enable-server="true"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116710    4095 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116715    4095 flags.go:64] FLAG: --event-burst="100"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116718    4095 flags.go:64] FLAG: --event-qps="50"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116721    4095 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116724    4095 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116727    4095 flags.go:64] FLAG: --eviction-hard=""
Nov 01 12:22:20.123480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116731    4095 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116734    4095 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116737    4095 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116740    4095 flags.go:64] FLAG: --eviction-soft=""
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116743    4095 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116746    4095 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116749    4095 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116752    4095 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116755    4095 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116758    4095 flags.go:64] FLAG: --fail-swap-on="true"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116761    4095 flags.go:64] FLAG: --feature-gates=""
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116765    4095 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116768    4095 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116771    4095 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116775    4095 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116780    4095 flags.go:64] FLAG: --healthz-port="10248"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116784    4095 flags.go:64] FLAG: --help="false"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116787    4095 flags.go:64] FLAG: --hostname-override="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116790    4095 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116793    4095 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116796    4095 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116800    4095 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116803    4095 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 01 12:22:20.124073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116806    4095 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116810    4095 flags.go:64] FLAG: --image-service-endpoint=""
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116812    4095 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116816    4095 flags.go:64] FLAG: --kube-api-burst="100"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116819    4095 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116822    4095 flags.go:64] FLAG: --kube-api-qps="50"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116824    4095 flags.go:64] FLAG: --kube-reserved=""
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116827    4095 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116830    4095 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116833    4095 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116836    4095 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116839    4095 flags.go:64] FLAG: --lock-file=""
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116841    4095 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116844    4095 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116847    4095 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116852    4095 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116855    4095 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116858    4095 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116860    4095 flags.go:64] FLAG: --logging-format="text"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116863    4095 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116867    4095 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116870    4095 flags.go:64] FLAG: --manifest-url=""
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116873    4095 flags.go:64] FLAG: --manifest-url-header=""
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116877    4095 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116881    4095 flags.go:64] FLAG: --max-open-files="1000000"
Nov 01 12:22:20.124638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116887    4095 flags.go:64] FLAG: --max-pods="110"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116890    4095 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116893    4095 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116896    4095 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116899    4095 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116902    4095 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116905    4095 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116908    4095 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116916    4095 flags.go:64] FLAG: --node-status-max-images="50"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116920    4095 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116923    4095 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116926    4095 flags.go:64] FLAG: --pod-cidr=""
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116928    4095 flags.go:64] FLAG: --pod-manifest-path=""
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116932    4095 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116935    4095 flags.go:64] FLAG: --pods-per-core="0"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116938    4095 flags.go:64] FLAG: --port="10250"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116941    4095 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116943    4095 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0098837b9375d32bb"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116947    4095 flags.go:64] FLAG: --qos-reserved=""
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116950    4095 flags.go:64] FLAG: --read-only-port="10255"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116953    4095 flags.go:64] FLAG: --register-node="true"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116955    4095 flags.go:64] FLAG: --register-with-taints=""
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116959    4095 flags.go:64] FLAG: --registry-burst="10"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116962    4095 flags.go:64] FLAG: --registry-qps="5"
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116965    4095 flags.go:64] FLAG: --reserved-cpus=""
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116981    4095 flags.go:64] FLAG: --reserved-memory=""
Nov 01 12:22:20.125315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116985    4095 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116988    4095 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116991    4095 flags.go:64] FLAG: --rotate-certificates="false"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116994    4095 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.116997    4095 flags.go:64] FLAG: --runonce="false"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117001    4095 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117005    4095 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117008    4095 flags.go:64] FLAG: --seccomp-default="false"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117011    4095 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117014    4095 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117017    4095 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117020    4095 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117023    4095 flags.go:64] FLAG: --storage-driver-password="root"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117026    4095 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117029    4095 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117032    4095 flags.go:64] FLAG: --storage-driver-user="root"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117035    4095 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117038    4095 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117041    4095 flags.go:64] FLAG: --system-cgroups=""
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117044    4095 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117049    4095 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117052    4095 flags.go:64] FLAG: --tls-cert-file=""
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117055    4095 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117058    4095 flags.go:64] FLAG: --tls-min-version=""
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117061    4095 flags.go:64] FLAG: --tls-private-key-file=""
Nov 01 12:22:20.125932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117064    4095 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117067    4095 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117070    4095 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117072    4095 flags.go:64] FLAG: --v="2"
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117076    4095 flags.go:64] FLAG: --version="false"
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117080    4095 flags.go:64] FLAG: --vmodule=""
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117084    4095 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117087    4095 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117213    4095 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117216    4095 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117219    4095 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117222    4095 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117226    4095 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117230    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117233    4095 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117236    4095 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117238    4095 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117241    4095 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117243    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117246    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117249    4095 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:20.126574 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117252    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117254    4095 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117257    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117260    4095 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117263    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117266    4095 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117268    4095 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117271    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117274    4095 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117276    4095 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117279    4095 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117281    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117284    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117286    4095 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117289    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117291    4095 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117294    4095 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117296    4095 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117299    4095 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117301    4095 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:20.127115 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117304    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117307    4095 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117310    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117313    4095 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117317    4095 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117320    4095 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117322    4095 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117325    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117327    4095 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117330    4095 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117333    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117335    4095 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117338    4095 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117340    4095 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117343    4095 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117346    4095 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117348    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117351    4095 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117354    4095 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:20.127590 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117356    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117359    4095 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117361    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117364    4095 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117367    4095 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117369    4095 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117372    4095 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117375    4095 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117377    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117380    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117382    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117384    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117388    4095 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117392    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117395    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117398    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117402    4095 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117406    4095 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117408    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:20.128067 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117411    4095 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117413    4095 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117416    4095 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117419    4095 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117421    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117424    4095 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117426    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117429    4095 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117431    4095 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117434    4095 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117437    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117440    4095 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117442    4095 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117445    4095 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117448    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117451    4095 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117453    4095 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117456    4095 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117458    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117461    4095 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:20.128534 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117464    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117466    4095 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117468    4095 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117472    4095 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117476    4095 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117479    4095 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117482    4095 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117485    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117487    4095 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117491    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117495    4095 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117498    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117501    4095 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117504    4095 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117507    4095 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.117531    4095 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:22:20.129339 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117572    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117577    4095 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117580    4095 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117583    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117586    4095 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117589    4095 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117592    4095 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117594    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117597    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117600    4095 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117602    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117605    4095 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117608    4095 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117610    4095 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117613    4095 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117615    4095 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117618    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117620    4095 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117623    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:20.130006 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117625    4095 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117628    4095 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117631    4095 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117633    4095 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117635    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117638    4095 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117641    4095 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117645    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117647    4095 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117650    4095 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117652    4095 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117655    4095 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117657    4095 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117660    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117662    4095 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117665    4095 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117667    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117670    4095 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117672    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:20.130653 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117675    4095 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117677    4095 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117680    4095 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117683    4095 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117685    4095 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117688    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117690    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117693    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117695    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117699    4095 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117702    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117706    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117709    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117712    4095 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117715    4095 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117717    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117719    4095 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117722    4095 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117725    4095 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:20.131189 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117727    4095 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117731    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117734    4095 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117736    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117739    4095 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117741    4095 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117744    4095 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117746    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117749    4095 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117751    4095 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117754    4095 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117756    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117759    4095 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117761    4095 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117764    4095 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117766    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117769    4095 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117772    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117774    4095 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:20.131787 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117777    4095 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117781    4095 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117784    4095 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117787    4095 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117789    4095 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117792    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117794    4095 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117797    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117800    4095 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117803    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117805    4095 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117807    4095 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117810    4095 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117812    4095 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117816    4095 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117819    4095 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117821    4095 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117824    4095 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117827    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117829    4095 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:20.132326 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117831    4095 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117834    4095 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117836    4095 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117839    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117841    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117844    4095 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117846    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117849    4095 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117852    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.117854    4095 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.124050    4095 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.124063    4095 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.124082    4095 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.124087    4095 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124145    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124150    4095 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124153    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124156    4095 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124160    4095 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124163    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:20.132813 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124166    4095 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124169    4095 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124172    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124175    4095 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124178    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124180    4095 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124183    4095 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124186    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124188    4095 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124191    4095 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124194    4095 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124197    4095 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124201    4095 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124203    4095 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124206    4095 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124208    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124212    4095 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124216    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124219    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124222    4095 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:20.133306 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124224    4095 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124227    4095 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124229    4095 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124232    4095 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124234    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124238    4095 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124241    4095 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124244    4095 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124246    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124249    4095 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124252    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124254    4095 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124257    4095 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124260    4095 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124262    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124265    4095 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124267    4095 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124270    4095 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124273    4095 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124275    4095 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:20.133816 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124278    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124281    4095 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124284    4095 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124287    4095 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124290    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124292    4095 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124295    4095 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124297    4095 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124300    4095 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124303    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124305    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124307    4095 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124310    4095 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124313    4095 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124315    4095 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124318    4095 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124320    4095 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124323    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124326    4095 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124328    4095 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:20.134493 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124331    4095 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124335    4095 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124339    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124342    4095 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124345    4095 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124347    4095 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124350    4095 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124352    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124355    4095 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124357    4095 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124360    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124362    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124365    4095 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124368    4095 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124371    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124374    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124376    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124379    4095 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124382    4095 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:20.135028 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124384    4095 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124387    4095 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124389    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124392    4095 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124394    4095 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124397    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124399    4095 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124402    4095 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124404    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124407    4095 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124409    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124412    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124415    4095 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124417    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124419    4095 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124422    4095 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124425    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124428    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124430    4095 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:20.135494 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124433    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124435    4095 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.124463    4095 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124601    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124606    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124611    4095 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124615    4095 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124618    4095 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124621    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124624    4095 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124628    4095 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124631    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124634    4095 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124637    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124640    4095 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:20.135963 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124643    4095 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124646    4095 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124648    4095 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124651    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124654    4095 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124657    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124660    4095 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124663    4095 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124666    4095 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124668    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124671    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124673    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124676    4095 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124678    4095 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124681    4095 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124683    4095 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124686    4095 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124688    4095 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124691    4095 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124693    4095 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:20.136409 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124696    4095 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124699    4095 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124701    4095 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124704    4095 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124706    4095 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124709    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124712    4095 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124715    4095 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124718    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124721    4095 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124723    4095 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124726    4095 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124728    4095 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124731    4095 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124733    4095 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124736    4095 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124738    4095 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124741    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124744    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124747    4095 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:20.137199 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124749    4095 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124752    4095 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124754    4095 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124757    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124759    4095 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124762    4095 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124764    4095 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124768    4095 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124772    4095 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124775    4095 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124778    4095 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124780    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124783    4095 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124785    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124788    4095 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124790    4095 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124792    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124795    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124797    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:20.137834 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124800    4095 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124803    4095 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124805    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124808    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124810    4095 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124813    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124815    4095 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124818    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124820    4095 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124823    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124825    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124828    4095 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124830    4095 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124833    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124835    4095 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124837    4095 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124840    4095 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124843    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124845    4095 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124847    4095 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:20.138545 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124850    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124852    4095 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124855    4095 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124857    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124860    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124862    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124864    4095 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124867    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124870    4095 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124872    4095 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124875    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124877    4095 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124880    4095 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124883    4095 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124888    4095 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.124910    4095 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 01 12:22:20.139256 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124941    4095 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124945    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124948    4095 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124952    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124954    4095 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124957    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124959    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124962    4095 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124964    4095 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124981    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124985    4095 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124988    4095 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124990    4095 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124992    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124995    4095 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.124998    4095 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125001    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125003    4095 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125005    4095 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125008    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 01 12:22:20.139650 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125011    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125014    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125016    4095 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125019    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125021    4095 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125024    4095 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125027    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125029    4095 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125032    4095 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125034    4095 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125040    4095 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125042    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125045    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125049    4095 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125053    4095 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125056    4095 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125059    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125061    4095 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125064    4095 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 01 12:22:20.140196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125066    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125069    4095 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125071    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125074    4095 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125077    4095 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125079    4095 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125082    4095 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125084    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125087    4095 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125089    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125092    4095 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125094    4095 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125097    4095 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125099    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125102    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125104    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125107    4095 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125109    4095 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125112    4095 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125114    4095 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 01 12:22:20.140675 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125117    4095 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125119    4095 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125122    4095 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125126    4095 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125135    4095 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125138    4095 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125140    4095 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125143    4095 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125146    4095 feature_gate.go:386] unrecognized feature gate: Example
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125148    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125151    4095 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125153    4095 feature_gate.go:386] unrecognized feature gate: Example2
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125156    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125158    4095 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125161    4095 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125164    4095 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125166    4095 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125169    4095 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125171    4095 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125174    4095 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 01 12:22:20.141241 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125176    4095 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125179    4095 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125182    4095 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125184    4095 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125187    4095 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125190    4095 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125192    4095 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125195    4095 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125199    4095 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125201    4095 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125204    4095 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125209    4095 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125212    4095 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125216    4095 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125218    4095 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125222    4095 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125225    4095 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125228    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125230    4095 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 01 12:22:20.141806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125233    4095 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125236    4095 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125238    4095 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125241    4095 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125243    4095 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125246    4095 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125248    4095 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:20.125251    4095 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.125298    4095 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.126280    4095 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.126328    4095 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.126504    4095 server.go:1014] "Starting client certificate rotation"
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.126649    4095 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.126696    4095 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-31 12:16:38 +0000 UTC" deadline="2028-06-03 13:21:44.012435677 +0000 UTC"
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.126724    4095 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13920h59m23.885713569s"
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.126944    4095 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 12:22:20.142280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.127959    4095 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 01 12:22:20.142669 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.129586    4095 log.go:25] "Validated CRI v1 runtime API"
Nov 01 12:22:20.142669 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.131067    4095 log.go:25] "Validated CRI v1 image API"
Nov 01 12:22:20.142669 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.131823    4095 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 01 12:22:20.142669 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.133205    4095 fs.go:135] Filesystem UUIDs: map[4007a117-6838-4a92-ba64-07d5882b1f0a:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 a6560e3c-2503-4125-842e-43411efd96ab:/dev/nvme0n1p3]
Nov 01 12:22:20.142786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.133222    4095 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0452ea431d5404e55d2961d22e965674db16147e36a067c7089e4d401140c341/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0452ea431d5404e55d2961d22e965674db16147e36a067c7089e4d401140c341/userdata/shm major:0 minor:201 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15d3a17c849abcd4ae6b211d7007cc23c4046eadea28261036cbb9139c1ecf02/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15d3a17c849abcd4ae6b211d7007cc23c4046eadea28261036cbb9139c1ecf02/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15fe03495c807090f05994076727b74d2a9b93b7082764faaf38c0cd24672c89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15fe03495c807090f05994076727b74d2a9b93b7082764faaf38c0cd24672c89/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1cf85aebf6b03fffe41cbb895bdca1535405fcaa66abc6353ef79fcef60a4515/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1cf85aebf6b03fffe41cbb895bdca1535405fcaa66abc6353ef79fcef60a4515/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a320ab6e458e4d86d522a8be08108efd6e2d46d8681f736235050b789bfd93b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a320ab6e458e4d86d522a8be08108efd6e2d46d8681f736235050b789bfd93b/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45eb50a7bf8353c6edbb15bab2fb8c6c7eefb817df10c46c59c819462787f810/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45eb50a7bf8353c6edbb15bab2fb8c6c7eefb817df10c46c59c819462787f810/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47f48264521834cc2f6b82b67ccc14735ec341f779a26dd48d6bfac50ba29392/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47f48264521834cc2f6b82b67ccc14735ec341f779a26dd48d6bfac50ba29392/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/58f62c0c26b1fe16431db461aa138a39f3160518ca3f4db4e24fe7fd8d8d5c2e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/58f62c0c26b1fe16431db461aa138a39f3160518ca3f4db4e24fe7fd8d8d5c2e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/63e6d3347c5a8c175c746bc2ee2f0d914839e78aba2355a4bd3fbb7dc93fc25d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/63e6d3347c5a8c175c746bc2ee2f0d914839e78aba2355a4bd3fbb7dc93fc25d/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7ba56b2e4284c5b7d4c3cd52d9ea62847f2a321dd3e1068c26ecce19f95258b3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7ba56b2e4284c5b7d4c3cd52d9ea62847f2a321dd3e1068c26ecce19f95258b3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8954c7579906fbf4e5536ed5a8948a443b2697ef0bf87ce1fc9dec9ebc9023e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8954c7579906fbf4e5536ed5a8948a443b2697ef0bf87ce1fc9dec9ebc9023e3/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e79eaadcd561dd01938bf27ef2210aaeb235fa9fcc6abd9d320ff97ec85ca9d3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e79eaadcd561dd01938bf27ef2210aaeb235fa9fcc6abd9d320ff97ec85ca9d3/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/04bb07a5-23a4-4095-a019-cb8a36e1bd63/volumes/kubernetes.io~projected/kube-api-access-drtsk:{mountpoint:/var/lib/kubelet/pods/04bb07a5-23a4-4095-a019-cb8a36e1bd63/volumes/kubernetes.io~projected/kube-api-access-drtsk major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6a342b-6f0b-4701-919e-d46c24861d27/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/0b6a342b-6f0b-4701-919e-d46c24861d27/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6a342b-6f0b-4701-919e-d46c24861d27/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/0b6a342b-6f0b-4701-919e-d46c24861d27/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0b6a342b-6f0b-4701-919e-d46c24861d27/volumes/kubernetes.io~projected/kube-api-access-l59hh:{mountpoint:/var/lib/kubelet/pods/0b6a342b-6f0b-4701-919e-d46c24861d27/volumes/kubernetes.io~projected/kube-api-access-l59hh major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/246aad42-5862-4e6e-af04-a7a9847f4ab5/volumes/kubernetes.io~projected/kube-api-access-g877r:{mountpoint:/var/lib/kubelet/pods/246aad42-5862-4e6e-af04-a7a9847f4ab5/volumes/kubernetes.io~projected/kube-api-access-g877r major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/267252e3-49bf-4478-90fb-2660c06b2b62/volumes/kubernetes.io~projected/kube-api-access-992n4:{mountpoint:/var/lib/kubelet/pods/267252e3-49bf-4478-90fb-2660c06b2b62/volumes/kubernetes.io~projected/kube-api-access-992n4 major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3a7a84c2-0167-4f45-82b8-d8da5f192fac/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/3a7a84c2-0167-4f45-82b8-d8da5f192fac/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/752a2f8b-cf54-4bf3-bd4d-39ba739cb627/volumes/kubernetes.io~projected/kube-api-access-cc7p7:{mountpoint:/var/lib/kubelet/pods/752a2f8b-cf54-4bf3-bd4d-39ba739cb627/volumes/kubernetes.io~projected/kube-api-access-cc7p7 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33/volumes/kubernetes.io~projected/kube-api-access-d2xl9:{mountpoint:/var/lib/kubelet/pods/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33/volumes/kubernetes.io~projected/kube-api-access-d2xl9 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a718c0bb-3ac8-432b-9b69-60e61375994f/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/a718c0bb-3ac8-432b-9b69-60e61375994f/volumes/kubernetes.io~secret/agent-certs major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866/volumes/kubernetes.io~projected/kube-api-access-k467p:{mountpoint:/var/lib/kubelet/pods/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866/volumes/kubernetes.io~projected/kube-api-access-k467p major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c76d9907-a77b-455f-9437-660fd451b40b/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c76d9907-a77b-455f-9437-660fd451b40b/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c76d9907-a77b-455f-9437-660fd451b40b/volumes/kubernetes.io~projected/kube-api-access-xhbsf:{mountpoint:/var/lib/kubelet/pods/c76d9907-a77b-455f-9437-660fd451b40b/volumes/kubernetes.io~projected/kube-api-access-xhbsf major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c76d9907-a77b-455f-9437-660fd451b40b/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c76d9907-a77b-455f-9437-660fd451b40b/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d19a543b-1191-4040-b996-15e710bbb1bf/volumes/kubernetes.io~projected/kube-api-access-8fck6:{mountpoint:/var/lib/kubelet/pods/d19a543b-1191-4040-b996-15e710bbb1bf/volumes/kubernetes.io~projected/kube-api-access-8fck6 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ed6f24b7-7844-4464-9325-e41004b17167/volumes/kubernetes.io~projected/kube-api-access-zm6p9:{mountpoint:/var/lib/kubelet/pods/ed6f24b7-7844-4464-9325-e41004b17167/volumes/kubernetes.io~projected/kube-api-access-zm6p9 major:0 minor:65 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/eff3ba2ed5d6397e1e9bd800cf33e1d2fe37f63a228f30b20d28a617ce489e1a/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d7b814a54345b7a16ece60e1729a8b9eb6df2fe499016025a1bacdc145189161/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/51ba37ea19deb299bb210a1b47eebba9c090610f9ca67c49f27b9786577a7a31/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/a513ab88737ee4e5347bc8f9137075a6f5fd7c7f98d9cce77ab57e32057535a2/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/d2da846072dd25771a9cc3da37edc442cbf5ce2109c10397cb5657e7ae518a00/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/97bacc09933ab8ae293ad0aed4ff1faa52a3dd2ac4d2c02bcde5a75ca0ada100/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/9c75b003c6bed2f23581a4abe2e8aaceab5d58abac8032416e66a63a2977488b/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c505d8cf31ef163de58eb8cc6bc9807534be93b8239dd6d7431a501b0d600c9c/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/42578f9e634c9b8928987ef1443b4a2fb8084a7c2bfc6e60ebfab0f0fac8cb08/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/959f9179dc2af15cb0c1a54f2c83453b2d07ec24f1b571f70f9dce0c4f2ccd89/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/b2d931f29355a3afb38a0cdacd65fe778a0f31c29773483ae70077b4959fcd90/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/40d6bc4c55b1f41e6f693eb982e622b115daa6cec6ffe168958e7ca05774dad0/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/e009de79f0a7f93c6228eefbef675b1273199fc566e755f068cc47d1cb69bbb5/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/d6c112d2ea74842cc0590f56957bb065f50bfbaced348bf117c61361875b2cbf/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/eb8822b1d9a83d903203e35e9a7fbbddae9bc5272bb55eba592f88b057151ee0/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/4a4175ed3a1be0bbd60b7159406f9e4cad370061c07bd2b27396f64f81a8853a/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/aaf93d4a0d30c29ec8b0bf7b4e4cb0366748c9019d0e3fd9039a750c5357c8d7/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/d67b7580cfef9621ab3393e6c24d5b9d27dce673cfd0cdba2c75fbb98c72a13e/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/021128e56a05ee006c3e91dcce77e137424ac69d439896e0ab3b4a5a2b5be6f1/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-184:{mountpoint:/var/lib/containers/storage/overlay/4dd2e2afc8a86fedbe43fb87220fa60cbabda6d3944fa724bb6d154b4c3c95c7/merged major:0 minor:184 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c87f9e85b4f1946b2827e548eab78d31ab6d1143d1a54c656c30b227c2a346ca/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-205:{mountpoint:/var/lib/containers/storage/overlay/062c69b9c3e50a1cd391293b2807f637b786be0c9754b703beaf0f0cb4e29f0e/merged major:0 minor:205 fsType:overlay blockSize:0} overlay_0-207:{mountpoint:/var/lib/containers/storage/overlay/9ab6e6a7f023ef1cb25fb67979cdc0a981da1befea745dba13e84ada311c4fbf/merged major:0 minor:207 fsType:overlay blockSize:0} overlay_0-209:{mountpoint:/var/lib/containers/storage/overlay/75543e60e574428505626afd7d082681fcfa6319a0354275a754849163d878f2/merged major:0 minor:209 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/079bdebe3b85c3ada8abd6e9eed00abc9260865aa69c3ae7bbf66d54eb40c1d5/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/0917bba19a7a8b68fbee9e6e0f7828e5cdaa04a80103a1e0ea7bd7e531891f8c/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/f77587e982ed0b39d53b6e49ff2fe523db868792799b7a265523eb3b9736cdc9/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/774321545bed158d64dc834f38c5c432d917fee2532a07d851e836cacc75ad18/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/5c72938fef51a4e38ba8ca6fa8d51daaf107502d24497a584ab8d6b3df45920a/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/325c32440e3615010838b2891f39b221c46b9d598a6d1a5e74a0283859b7ad82/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ca4237314c1579c083d554b0bb263b0b1b80ad0dc396d9e1985be749c45bea3b/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/75e930eb57e893d2146a64efe52ade395800089265255839e4c71595a794d107/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/504b4d1e1133008c5ac90811ff7acf6d03080fa02c6dde05f1419c297b2a043a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/1ca376c1d4101766074908dcb57ed8a372227333cf0f43c849bdde28e2ab49a1/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/b606034ff855257f6822563ece12ac811db9f35b02869cbe45e45dc060913a96/merged major:0 minor:96 fsType:overlay blockSize:0}]
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.139456    4095 manager.go:217] Machine: {Timestamp:2026-11-01 12:22:20.139004542 +0000 UTC m=+0.056800199 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099709 MemoryCapacity:33164341248 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22eb1636ee97c907452bcfaf8f6d15 SystemUUID:ec22eb16-36ee-97c9-0745-2bcfaf8f6d15 BootID:981d3d92-f884-41ad-a9cd-22b9a28ef5f9 Filesystems:[{Device:overlay_0-184 DeviceMajor:0 DeviceMinor:184 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a320ab6e458e4d86d522a8be08108efd6e2d46d8681f736235050b789bfd93b/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-209 DeviceMajor:0 DeviceMinor:209 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-205 DeviceMajor:0 DeviceMinor:205 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/63e6d3347c5a8c175c746bc2ee2f0d914839e78aba2355a4bd3fbb7dc93fc25d/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c76d9907-a77b-455f-9437-660fd451b40b/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/267252e3-49bf-4478-90fb-2660c06b2b62/volumes/kubernetes.io~projected/kube-api-access-992n4 DeviceMajor:0 DeviceMinor:61 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/246aad42-5862-4e6e-af04-a7a9847f4ab5/volumes/kubernetes.io~projected/kube-api-access-g877r DeviceMajor:0 DeviceMinor:63 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/58f62c0c26b1fe16431db461aa138a39f3160518ca3f4db4e24fe7fd8d8d5c2e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8954c7579906fbf4e5536ed5a8948a443b2697ef0bf87ce1fc9dec9ebc9023e3/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15fe03495c807090f05994076727b74d2a9b93b7082764faaf38c0cd24672c89/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6a342b-6f0b-4701-919e-d46c24861d27/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:58 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/c76d9907-a77b-455f-9437-660fd451b40b/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:55 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866/volumes/kubernetes.io~projected/kube-api-access-k467p DeviceMajor:0 DeviceMinor:197 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582168576 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/a718c0bb-3ac8-432b-9b69-60e61375994f/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:57 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/04bb07a5-23a4-4095-a019-cb8a36e1bd63/volumes/kubernetes.io~projected/kube-api-access-drtsk DeviceMajor:0 DeviceMinor:67 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1cf85aebf6b03fffe41cbb895bdca1535405fcaa66abc6353ef79fcef60a4515/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7ba56b2e4284c5b7d4c3cd52d9ea62847f2a321dd3e1068c26ecce19f95258b3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3a7a84c2-0167-4f45-82b8-d8da5f192fac/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:45 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/d19a543b-1191-4040-b996-15e710bbb1bf/volumes/kubernetes.io~projected/kube-api-access-8fck6 DeviceMajor:0 DeviceMinor:59 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/ed6f24b7-7844-4464-9325-e41004b17167/volumes/kubernetes.io~projected/kube-api-access-zm6p9 DeviceMajor:0 DeviceMinor:65 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0452ea431d5404e55d2961d22e965674db16147e36a067c7089e4d401140c341/userdata/shm DeviceMajor:0 DeviceMinor:201 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15d3a17c849abcd4ae6b211d7007cc23c4046eadea28261036cbb9139c1ecf02/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33/volumes/kubernetes.io~projected/kube-api-access-d2xl9 DeviceMajor:0 DeviceMinor:66 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47f48264521834cc2f6b82b67ccc14735ec341f779a26dd48d6bfac50ba29392/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45eb50a7bf8353c6edbb15bab2fb8c6c7eefb817df10c46c59c819462787f810/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-207 DeviceMajor:0 DeviceMinor:207 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6a342b-6f0b-4701-919e-d46c24861d27/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/c76d9907-a77b-455f-9437-660fd451b40b/volumes/kubernetes.io~projected/kube-api-access-xhbsf DeviceMajor:0 DeviceMinor:60 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/var/lib/kubelet/pods/0b6a342b-6f0b-4701-919e-d46c24861d27/volumes/kubernetes.io~projected/kube-api-access-l59hh DeviceMajor:0 DeviceMinor:62 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632869888 Type:vfs Inodes:819200 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582172672 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/752a2f8b-cf54-4bf3-bd4d-39ba739cb627/volumes/kubernetes.io~projected/kube-api-access-cc7p7 DeviceMajor:0 DeviceMinor:64 Capacity:31985741824 Type:vfs Inodes:4048381 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e79eaadcd561dd01938bf27ef2210aaeb235fa9fcc6abd9d320ff97ec85ca9d3/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048381 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:0452ea431d5404e MacAddress:e6:c7:62:8a:79:95 Speed:10000 Mtu:8901} {Name:1cf85aebf6b03ff MacAddress:f2:1d:a6:79:d4:8a Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:f6:30:d6:cd Speed:0 Mtu:9001} {Name:br-int MacAddress:b6:0b:5f:76:01:e4 Speed:0 Mtu:8901} {Name:ens5 MacAddress:06:ff:f6:30:d6:cd Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:fa:f8:e8:3e:f9:f3 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:42:af:11:df:1f:a0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164341248 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.139681    4095 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.139751    4095 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140004    4095 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140022    4095 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-140-168.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140138    4095 topology_manager.go:143] "Creating topology manager with none policy"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140146    4095 container_manager_linux.go:311] "Creating device plugin manager"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140155    4095 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140173    4095 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140246    4095 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140383    4095 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140473    4095 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140505    4095 kubelet.go:493] "Attempting to sync node with API server"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140514    4095 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140523    4095 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140531    4095 kubelet.go:405] "Adding apiserver pod source"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.140539    4095 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.141153    4095 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.141165    4095 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 12:22:20.142912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.141577    4095 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 01 12:22:20.143586 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143193    4095 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 01 12:22:20.143586 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143201    4095 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 01 12:22:20.143586 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143285    4095 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 01 12:22:20.143586 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143471    4095 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 01 12:22:20.143586 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143497    4095 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 01 12:22:20.143586 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143574    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 01 12:22:20.143586 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143585    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143591    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143597    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143603    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143608    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143614    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143619    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143627    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143632    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143641    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143650    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143672    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 01 12:22:20.143754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.143678    4095 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 01 12:22:20.146782 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.146770    4095 server.go:1263] "Started kubelet"
Nov 01 12:22:20.146858 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.146832    4095 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 01 12:22:20.146920 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.146867    4095 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 01 12:22:20.146979 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.146942    4095 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 01 12:22:20.147223 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.147201    4095 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 01 12:22:20.147589 ip-10-0-140-168 systemd[1]: Started Kubernetes Kubelet.
Nov 01 12:22:20.147878 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.147861    4095 server.go:317] "Adding debug handlers to kubelet server"
Nov 01 12:22:20.152986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.151947    4095 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 01 12:22:20.152986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.152221    4095 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 01 12:22:20.152986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.152230    4095 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 01 12:22:20.152986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.152243    4095 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 01 12:22:20.152986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.152388    4095 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-31 12:16:40 +0000 UTC" deadline="2028-07-07 22:57:15.308926172 +0000 UTC"
Nov 01 12:22:20.152986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.152420    4095 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14746h34m55.156509841s"
Nov 01 12:22:20.152986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.152423    4095 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 01 12:22:20.152986 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:20.152588    4095 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154049    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="246aad42-5862-4e6e-af04-a7a9847f4ab5" volumeName="kubernetes.io/configmap/246aad42-5862-4e6e-af04-a7a9847f4ab5-serviceca" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154087    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="267252e3-49bf-4478-90fb-2660c06b2b62" volumeName="kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154101    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33" volumeName="kubernetes.io/configmap/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-iptables-alerter-script" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154115    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04bb07a5-23a4-4095-a019-cb8a36e1bd63" volumeName="kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cni-binary-copy" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154129    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04bb07a5-23a4-4095-a019-cb8a36e1bd63" volumeName="kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-daemon-config" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154141    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="246aad42-5862-4e6e-af04-a7a9847f4ab5" volumeName="kubernetes.io/projected/246aad42-5862-4e6e-af04-a7a9847f4ab5-kube-api-access-g877r" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154151    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="752a2f8b-cf54-4bf3-bd4d-39ba739cb627" volumeName="kubernetes.io/projected/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-kube-api-access-cc7p7" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154163    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866" volumeName="kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154195    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c76d9907-a77b-455f-9437-660fd451b40b" volumeName="kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-env-overrides" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154204    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c76d9907-a77b-455f-9437-660fd451b40b" volumeName="kubernetes.io/secret/c76d9907-a77b-455f-9437-660fd451b40b-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154211    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d19a543b-1191-4040-b996-15e710bbb1bf" volumeName="kubernetes.io/projected/d19a543b-1191-4040-b996-15e710bbb1bf-kube-api-access-8fck6" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154219    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="267252e3-49bf-4478-90fb-2660c06b2b62" volumeName="kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-binary-copy" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154226    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="267252e3-49bf-4478-90fb-2660c06b2b62" volumeName="kubernetes.io/projected/267252e3-49bf-4478-90fb-2660c06b2b62-kube-api-access-992n4" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154235    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33" volumeName="kubernetes.io/projected/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-kube-api-access-d2xl9" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154251    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c76d9907-a77b-455f-9437-660fd451b40b" volumeName="kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-config" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154259    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c76d9907-a77b-455f-9437-660fd451b40b" volumeName="kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-script-lib" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154266    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="267252e3-49bf-4478-90fb-2660c06b2b62" volumeName="kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154273    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3a7a84c2-0167-4f45-82b8-d8da5f192fac" volumeName="kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154280    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a718c0bb-3ac8-432b-9b69-60e61375994f" volumeName="kubernetes.io/configmap/a718c0bb-3ac8-432b-9b69-60e61375994f-konnectivity-ca" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154287    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a718c0bb-3ac8-432b-9b69-60e61375994f" volumeName="kubernetes.io/secret/a718c0bb-3ac8-432b-9b69-60e61375994f-agent-certs" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154295    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c76d9907-a77b-455f-9437-660fd451b40b" volumeName="kubernetes.io/projected/c76d9907-a77b-455f-9437-660fd451b40b-kube-api-access-xhbsf" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154303    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed6f24b7-7844-4464-9325-e41004b17167" volumeName="kubernetes.io/empty-dir/ed6f24b7-7844-4464-9325-e41004b17167-tmp-dir" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154310    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ed6f24b7-7844-4464-9325-e41004b17167" volumeName="kubernetes.io/projected/ed6f24b7-7844-4464-9325-e41004b17167-kube-api-access-zm6p9" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154322    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="04bb07a5-23a4-4095-a019-cb8a36e1bd63" volumeName="kubernetes.io/projected/04bb07a5-23a4-4095-a019-cb8a36e1bd63-kube-api-access-drtsk" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154333    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6a342b-6f0b-4701-919e-d46c24861d27" volumeName="kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-tuned" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154341    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6a342b-6f0b-4701-919e-d46c24861d27" volumeName="kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-tmp" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154349    4095 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0b6a342b-6f0b-4701-919e-d46c24861d27" volumeName="kubernetes.io/projected/0b6a342b-6f0b-4701-919e-d46c24861d27-kube-api-access-l59hh" seLinuxMountContext=""
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154356    4095 reconstruct.go:97] "Volume reconstruction finished"
Nov 01 12:22:20.154504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.154361    4095 reconciler.go:29] "Reconciler: start to sync state"
Nov 01 12:22:20.155696 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.155528    4095 factory.go:55] Registering systemd factory
Nov 01 12:22:20.155696 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.155543    4095 factory.go:223] Registration of the systemd container factory successfully
Nov 01 12:22:20.155760 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.155747    4095 factory.go:153] Registering CRI-O factory
Nov 01 12:22:20.155760 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.155760    4095 factory.go:223] Registration of the crio container factory successfully
Nov 01 12:22:20.155855 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.155805    4095 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 01 12:22:20.155855 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.155822    4095 factory.go:103] Registering Raw factory
Nov 01 12:22:20.155855 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.155833    4095 manager.go:1199] Started watching for new ooms in manager
Nov 01 12:22:20.156270 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.156257    4095 manager.go:319] Starting recovery of all containers
Nov 01 12:22:20.163509 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.163463    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:22:20.163509 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.163498    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:22:20.180209 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.180168    4095 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 01 12:22:20.181362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.181340    4095 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 01 12:22:20.181443 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.181369    4095 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 01 12:22:20.181443 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.181391    4095 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 01 12:22:20.181443 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:20.181430    4095 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 01 12:22:20.188060 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.188032    4095 generic.go:356] "Generic (PLEG): container finished" podID="2ee0d9d2202565b503aec356e54ab8fe" containerID="2dce942190859e7ef07106b65d9f4b4272fc12997fd95ce9c532ad8a1b1032ef" exitCode=0
Nov 01 12:22:20.197893 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.197872    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:22:20.198354 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.198326    4095 generic.go:356] "Generic (PLEG): container finished" podID="c76d9907-a77b-455f-9437-660fd451b40b" containerID="8f0d60fb59addc8e6846e68ad44b5b7a37849f102611e10125629173a067dfb2" exitCode=1
Nov 01 12:22:20.208666 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.208647    4095 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="4b2851e53148e87e11af6752f3e565eb4884c387376dd3c36510373abd4a5712" exitCode=0
Nov 01 12:22:20.208666 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.208662    4095 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="abe54d15ad0852976f83f40b1b7f3db3c49943a3524eddf203dcd86f179c314f" exitCode=0
Nov 01 12:22:20.208666 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.208671    4095 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="0bb59d2f16b886fa9cc2dbc520ed260bf61fb763f61bfc3833fb5b16dbef2810" exitCode=0
Nov 01 12:22:20.208666 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.208676    4095 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="c75c6aae2760ca55ec2675f7a9b917119eed2652c553b3dd4dd6b238d4457654" exitCode=0
Nov 01 12:22:20.208900 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.208684    4095 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="1413e38a298ea2b09f78dd20eb139c25fdd1c2070e63b40a5e992edf050a1051" exitCode=0
Nov 01 12:22:20.208900 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.208693    4095 generic.go:356] "Generic (PLEG): container finished" podID="267252e3-49bf-4478-90fb-2660c06b2b62" containerID="70b5ba8c9fa4487aaaeb93c1da024995dc2732d44cd6f39627ca36ee24f002bc" exitCode=0
Nov 01 12:22:20.211514 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.211499    4095 manager.go:324] Recovery completed
Nov 01 12:22:20.224389 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.224372    4095 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:20.225606 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.225591    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:20.225658 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.225619    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:20.225658 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.225629    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:20.226795 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.226780    4095 cpu_manager.go:226] "Starting" policy="none"
Nov 01 12:22:20.226795 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.226792    4095 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:20.226870 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.226795    4095 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 01 12:22:20.226902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.226880    4095 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 01 12:22:20.227025 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227008    4095 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 01 12:22:20.227071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227026    4095 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 01 12:22:20.227071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227043    4095 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 01 12:22:20.227071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227049    4095 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 01 12:22:20.227071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227055    4095 policy_none.go:50] "Start"
Nov 01 12:22:20.227071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227064    4095 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 01 12:22:20.227299 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227077    4095 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 01 12:22:20.227613 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227598    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:20.227660 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227625    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:20.227660 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.227635    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:20.228208 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.228193    4095 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 01 12:22:20.228208 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.228209    4095 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 01 12:22:20.228297 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.228215    4095 policy_none.go:44] "Start"
Nov 01 12:22:20.232266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.232233    4095 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 01 12:22:20.237739 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.237728    4095 manager.go:341] "Starting Device Plugin manager"
Nov 01 12:22:20.237821 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:20.237749    4095 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 01 12:22:20.237821 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.237758    4095 server.go:87] "Starting device plugin registration server"
Nov 01 12:22:20.238039 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.238028    4095 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 01 12:22:20.238086 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.238041    4095 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 01 12:22:20.238145 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.238120    4095 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 01 12:22:20.238226 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.238200    4095 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 01 12:22:20.238276 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.238238    4095 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 01 12:22:20.238276 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.238245    4095 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 01 12:22:20.238376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.238300    4095 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-01T12:22:20.238223267Z","UUID":"e3a5ab78-046f-4853-9ddd-19ebb8d8d711","Handler":null,"Name":"","Endpoint":""}
Nov 01 12:22:20.239392 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:20.239326    4095 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 01 12:22:20.239392 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:20.239362    4095 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-140-168.us-west-2.compute.internal\" not found"
Nov 01 12:22:20.239532 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.239480    4095 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 01 12:22:20.239532 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.239512    4095 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 01 12:22:20.282080 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.281531    4095 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"]
Nov 01 12:22:20.282216 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.282128    4095 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:20.283040 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.283023    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:20.283120 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.283053    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:20.283120 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.283065    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:20.283280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.283271    4095 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:20.283420 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.283407    4095 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:20.283954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.283935    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:20.283954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.283955    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:20.284070 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.283980    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:20.284070 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284005    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:20.284070 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284023    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:20.284070 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284032    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:20.284206 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:20.284170    4095 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found" node="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.284206 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284161    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"2ee0d9d2202565b503aec356e54ab8fe","Type":"ContainerStarted","Data":"9822ffca28abb0c2a170ebc5332d71ef2d8b393298241ff8bb177baa2f774661"}
Nov 01 12:22:20.284206 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284202    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"2ee0d9d2202565b503aec356e54ab8fe","Type":"ContainerDied","Data":"2dce942190859e7ef07106b65d9f4b4272fc12997fd95ce9c532ad8a1b1032ef"}
Nov 01 12:22:20.284337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284221    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"2ee0d9d2202565b503aec356e54ab8fe","Type":"ContainerStarted","Data":"15fe03495c807090f05994076727b74d2a9b93b7082764faaf38c0cd24672c89"}
Nov 01 12:22:20.284337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284234    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"7042c7ca4bb39d530e5b25fb56952b15","Type":"ContainerStarted","Data":"0a82b62c9f5210a21d4d94805f64e9e63f6a9c2d9f94fed124dc2d49c9d7510d"}
Nov 01 12:22:20.284337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284246    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal" event={"ID":"7042c7ca4bb39d530e5b25fb56952b15","Type":"ContainerStarted","Data":"8954c7579906fbf4e5536ed5a8948a443b2697ef0bf87ce1fc9dec9ebc9023e3"}
Nov 01 12:22:20.284337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284281    4095 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:20.284944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284930    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:20.285039 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284957    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:20.285039 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.284985    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:20.285153 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:20.285141    4095 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-140-168.us-west-2.compute.internal\" not found" node="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.336618 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.336590    4095 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:22:20.336746 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.336594    4095 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 01 12:22:20.338558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.338543    4095 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 01 12:22:20.339082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.339066    4095 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:22:20.339609 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.339594    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Nov 01 12:22:20.339656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.339622    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Nov 01 12:22:20.339656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.339633    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeHasSufficientPID"
Nov 01 12:22:20.339734 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.339661    4095 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.353823 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.353807    4095 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.384226 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.384202    4095 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:22:20.387333 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.387308    4095 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 01 12:22:20.431535 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.431494    4095 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:22:20.431535 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:20.431553    4095 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.431755 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.431571    4095 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.437787 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.437767    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.437838 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.437793    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.437838 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.437820    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7042c7ca4bb39d530e5b25fb56952b15-config\") pod \"kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"7042c7ca4bb39d530e5b25fb56952b15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.480785 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.480761    4095 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.480854 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.480832    4095 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.481398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.481381    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeNotReady"
Nov 01 12:22:20.481488 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.481417    4095 setters.go:546] "Node became not ready" node="ip-10-0-140-168.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-01T12:22:20Z","lastTransitionTime":"2026-11-01T12:22:20Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Nov 01 12:22:20.488533 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.488512    4095 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:22:20.488647 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:20.488565    4095 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.493610 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.493589    4095 nodeinfomanager.go:471] existing CSINode "ip-10-0-140-168.us-west-2.compute.internal" is owned by different node (oldNodeID="9ce9f195-aceb-470f-830a-e84e3f324671", newNodeID=""), cleaning up...
Nov 01 12:22:20.538803 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.538743    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.538803 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.538775    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.538803 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.538795    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7042c7ca4bb39d530e5b25fb56952b15-config\") pod \"kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"7042c7ca4bb39d530e5b25fb56952b15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.538992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.538855    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/7042c7ca4bb39d530e5b25fb56952b15-config\") pod \"kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"7042c7ca4bb39d530e5b25fb56952b15\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.538992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.538855    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:20.538992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:20.538869    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ee0d9d2202565b503aec356e54ab8fe-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" (UID: \"2ee0d9d2202565b503aec356e54ab8fe\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:21.141642 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.141602    4095 apiserver.go:52] "Watching apiserver"
Nov 01 12:22:21.213090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.213064    4095 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:21.213233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.213184    4095 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:21.240402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.240374    4095 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 01 12:22:21.240764 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.240746    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-qtgbr","openshift-multus/network-metrics-daemon-dlcrd","openshift-ovn-kubernetes/ovnkube-node-hpzwg","kube-system/konnectivity-agent-nsn56","openshift-image-registry/node-ca-ld2ml","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-gnbj7","kube-system/global-pull-secret-syncer-5pg82","openshift-cluster-node-tuning-operator/tuned-vw5xj","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx","openshift-dns/node-resolver-wd6jh","openshift-multus/multus-86859","openshift-network-operator/iptables-alerter-bchgz","kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"]
Nov 01 12:22:21.243639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.243619    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:21.253752 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.253735    4095 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 01 12:22:21.270456 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.270439    4095 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:22:21.270506 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:21.270491    4095 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:21.270993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.270965    4095 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 01 12:22:21.271034 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:21.271011    4095 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-140-168.us-west-2.compute.internal"
Nov 01 12:22:21.294072 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.294047    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:21.294175 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.294139    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 01 12:22:21.294288 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.294267    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 01 12:22:21.294478 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.294458    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-phch9\""
Nov 01 12:22:21.294544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.294477    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 01 12:22:21.294601 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.294552    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-n2z6g\""
Nov 01 12:22:21.294787 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.294759    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:22:21.295057 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295027    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 01 12:22:21.295244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295205    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 01 12:22:21.295244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295236    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 01 12:22:21.295376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295271    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 01 12:22:21.295376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295341    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 01 12:22:21.295465 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295452    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-qrhfv\""
Nov 01 12:22:21.295509 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295489    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 01 12:22:21.295626 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295605    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 01 12:22:21.295719 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295703    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 01 12:22:21.295795 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295772    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-7twmh\""
Nov 01 12:22:21.295898 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295885    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 01 12:22:21.296003 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.295988    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-9t2fc\""
Nov 01 12:22:21.296081 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.296069    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 01 12:22:21.296162 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.296151    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 01 12:22:21.296239 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.296224    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 01 12:22:21.296391 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.296372    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 01 12:22:21.296483 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.296468    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zhblh\""
Nov 01 12:22:21.296569 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.296554    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-9cl4q\""
Nov 01 12:22:21.342265 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342234    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:21.342265 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342264    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.342415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342284    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.342415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342301    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-kubernetes\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.342415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342317    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-lib-modules\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.342415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342381    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-conf-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.342615 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342435    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-log-socket\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.342615 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342481    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.342615 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342517    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-script-lib\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.342615 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342549    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.342777 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342617    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc7p7\" (UniqueName: \"kubernetes.io/projected/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-kube-api-access-cc7p7\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:21.342777 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342673    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fck6\" (UniqueName: \"kubernetes.io/projected/d19a543b-1191-4040-b996-15e710bbb1bf-kube-api-access-8fck6\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.342777 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342758    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-netns\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.342920 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342860    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-config\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.342957 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342938    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a718c0bb-3ac8-432b-9b69-60e61375994f-konnectivity-ca\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:22:21.343011 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342955    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/246aad42-5862-4e6e-af04-a7a9847f4ab5-host\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:22:21.343011 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.342985    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/246aad42-5862-4e6e-af04-a7a9847f4ab5-serviceca\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:22:21.343100 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343008    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-k8s-cni-cncf-io\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.343100 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343034    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.343100 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343093    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c76d9907-a77b-455f-9437-660fd451b40b-ovn-node-metrics-cert\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.343231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343146    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-cnibin\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.343231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343169    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-config\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.343312 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343233    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a718c0bb-3ac8-432b-9b69-60e61375994f-agent-certs\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:22:21.343312 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343302    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-host\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.343404 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343345    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.343404 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343384    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-iptables-alerter-script\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:22:21.343503 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343417    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-node-log\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.343571 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343498    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-env-overrides\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.343571 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343510    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/a718c0bb-3ac8-432b-9b69-60e61375994f-agent-certs\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:22:21.343571 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343531    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cnibin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.343709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343570    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-dbus\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:21.343709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343590    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-system-cni-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.343709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343605    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-system-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.343709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343602    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c76d9907-a77b-455f-9437-660fd451b40b-ovn-node-metrics-cert\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.343709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343617    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-iptables-alerter-script\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:22:21.343709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343656    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2xl9\" (UniqueName: \"kubernetes.io/projected/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-kube-api-access-d2xl9\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:22:21.343709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343676    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-slash\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.343709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343705    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-env-overrides\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.343709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343710    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-var-lib-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343749    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-sys\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343769    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l59hh\" (UniqueName: \"kubernetes.io/projected/0b6a342b-6f0b-4701-919e-d46c24861d27-kube-api-access-l59hh\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343776    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/246aad42-5862-4e6e-af04-a7a9847f4ab5-serviceca\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343795    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343814    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-host-slash\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343829    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-systemd-units\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343844    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-etc-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343864    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-tmp\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343931    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-tmp\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.343991    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-device-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344013    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-sys-fs\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344028    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6f24b7-7844-4464-9325-e41004b17167-tmp-dir\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344042    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-multus\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344056    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-systemd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.344078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344071    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-binary-copy\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344123    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed6f24b7-7844-4464-9325-e41004b17167-tmp-dir\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344172    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysconfig\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344221    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-conf\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344248    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-kubelet\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344261    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-binary-copy\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344269    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-multus-certs\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344302    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-var-lib-kubelet\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344328    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-kubelet-config\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344372    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344390    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhbsf\" (UniqueName: \"kubernetes.io/projected/c76d9907-a77b-455f-9437-660fd451b40b-kube-api-access-xhbsf\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344405    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-hostroot\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344418    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-modprobe-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344432    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-kubelet\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344470    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-992n4\" (UniqueName: \"kubernetes.io/projected/267252e3-49bf-4478-90fb-2660c06b2b62-kube-api-access-992n4\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344508    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g877r\" (UniqueName: \"kubernetes.io/projected/246aad42-5862-4e6e-af04-a7a9847f4ab5-kube-api-access-g877r\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344527    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-socket-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344542    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cni-binary-copy\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344565    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-socket-dir-parent\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.344639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344596    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-run\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344660    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-tuned\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344684    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed6f24b7-7844-4464-9325-e41004b17167-hosts-file\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344707    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zm6p9\" (UniqueName: \"kubernetes.io/projected/ed6f24b7-7844-4464-9325-e41004b17167-kube-api-access-zm6p9\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344739    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-bin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344772    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-tuned\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344798    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-daemon-config\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344809    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cni-binary-copy\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344854    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-ovn\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344890    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344917    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.344947    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-netns\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345031    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-bin\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345059    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-netd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345086    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345113    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/3a7a84c2-0167-4f45-82b8-d8da5f192fac-original-pull-secret\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345131    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345140    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345168    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-systemd\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345183    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-etc-kubernetes\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345197    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-registration-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.345351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.345214    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drtsk\" (UniqueName: \"kubernetes.io/projected/04bb07a5-23a4-4095-a019-cb8a36e1bd63-kube-api-access-drtsk\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.350957 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.350939    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 01 12:22:21.350957 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.350951    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 01 12:22:21.351076 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.350951    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 01 12:22:21.351076 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.351011    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 01 12:22:21.351076 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.351066    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 01 12:22:21.351197 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.351070    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 01 12:22:21.351197 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.351147    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 01 12:22:21.351197 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.351149    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 01 12:22:21.351197 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.351166    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 01 12:22:21.351197 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.351148    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 01 12:22:21.351336 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.351208    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 01 12:22:21.353163 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.353147    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/267252e3-49bf-4478-90fb-2660c06b2b62-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.353218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.353180    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c76d9907-a77b-455f-9437-660fd451b40b-ovnkube-script-lib\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.353287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.353272    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/a718c0bb-3ac8-432b-9b69-60e61375994f-konnectivity-ca\") pod \"konnectivity-agent-nsn56\" (UID: \"a718c0bb-3ac8-432b-9b69-60e61375994f\") " pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:22:21.355716 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.355699    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-daemon-config\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.393402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.393352    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 01 12:22:21.393479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.393466    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 01 12:22:21.393643 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.393626    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-nsq77\""
Nov 01 12:22:21.393693 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.393632    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-sfjvd\""
Nov 01 12:22:21.393693 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.393683    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-zhf7p\""
Nov 01 12:22:21.393787 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.393744    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzkjs\""
Nov 01 12:22:21.393827 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.393814    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-z9pcl\""
Nov 01 12:22:21.395986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.395954    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc7p7\" (UniqueName: \"kubernetes.io/projected/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-kube-api-access-cc7p7\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:21.396065 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.396049    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k467p\" (UniqueName: \"kubernetes.io/projected/c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866-kube-api-access-k467p\") pod \"network-check-target-gnbj7\" (UID: \"c13b7ab6-ee4e-4128-8e8a-1f4d7fe3c866\") " pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:21.396188 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.396120    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fck6\" (UniqueName: \"kubernetes.io/projected/d19a543b-1191-4040-b996-15e710bbb1bf-kube-api-access-8fck6\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.396713 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.396694    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l59hh\" (UniqueName: \"kubernetes.io/projected/0b6a342b-6f0b-4701-919e-d46c24861d27-kube-api-access-l59hh\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.396747 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.396712    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2xl9\" (UniqueName: \"kubernetes.io/projected/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-kube-api-access-d2xl9\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:22:21.397377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.397358    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-drtsk\" (UniqueName: \"kubernetes.io/projected/04bb07a5-23a4-4095-a019-cb8a36e1bd63-kube-api-access-drtsk\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.397440 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.397391    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-992n4\" (UniqueName: \"kubernetes.io/projected/267252e3-49bf-4478-90fb-2660c06b2b62-kube-api-access-992n4\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.397512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.397494    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g877r\" (UniqueName: \"kubernetes.io/projected/246aad42-5862-4e6e-af04-a7a9847f4ab5-kube-api-access-g877r\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:22:21.397545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.397514    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zm6p9\" (UniqueName: \"kubernetes.io/projected/ed6f24b7-7844-4464-9325-e41004b17167-kube-api-access-zm6p9\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:22:21.398151 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.398135    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhbsf\" (UniqueName: \"kubernetes.io/projected/c76d9907-a77b-455f-9437-660fd451b40b-kube-api-access-xhbsf\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.445553 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445530    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-node-log\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.445639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445560    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cnibin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.445639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445577    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-dbus\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:21.445639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445598    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-system-cni-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.445639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445620    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-system-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445642    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-slash\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445651    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-cnibin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445665    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-var-lib-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445670    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-system-cni-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445693    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-system-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445675    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-dbus\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445719    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-sys\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445664    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-node-log\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445746    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445701    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-var-lib-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445762    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-host-slash\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445771    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-sys\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445777    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-systemd-units\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.445789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445790    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445799    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-systemd-units\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445698    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-slash\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445800    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-etc-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445828    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-device-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445830    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33-host-slash\") pod \"iptables-alerter-bchgz\" (UID: \"a5eacd3c-0cc5-4ec9-a161-6d27ba8d3e33\") " pod="openshift-network-operator/iptables-alerter-bchgz"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445843    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-sys-fs\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445856    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-etc-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445859    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-multus\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445876    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-systemd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445898    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-device-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445913    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-multus\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445900    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-systemd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445926    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-sys-fs\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.445966    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysconfig\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446010    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-conf\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446053    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysconfig\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446073    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-kubelet\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446089    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-multus-certs\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446105    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-var-lib-kubelet\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446120    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-kubelet\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446125    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-kubelet-config\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446122    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-multus-certs\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446133    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-conf\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446151    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-var-lib-kubelet\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446154    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446175    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-hostroot\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446188    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/3a7a84c2-0167-4f45-82b8-d8da5f192fac-kubelet-config\") pod \"global-pull-secret-syncer-5pg82\" (UID: \"3a7a84c2-0167-4f45-82b8-d8da5f192fac\") " pod="kube-system/global-pull-secret-syncer-5pg82"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446212    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-modprobe-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446211    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-openvswitch\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446235    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-kubelet\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446257    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-socket-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446276    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-socket-dir-parent\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446284    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-kubelet\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446236    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-hostroot\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446309    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-run\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446313    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-modprobe-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446327    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed6f24b7-7844-4464-9325-e41004b17167-hosts-file\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446330    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-socket-dir-parent\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446337    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-socket-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446344    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-bin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446360    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ed6f24b7-7844-4464-9325-e41004b17167-hosts-file\") pod \"node-resolver-wd6jh\" (UID: \"ed6f24b7-7844-4464-9325-e41004b17167\") " pod="openshift-dns/node-resolver-wd6jh"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446359    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-run\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446370    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-ovn\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.446377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446399    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-netns\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446403    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-run-ovn\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446424    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-bin\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446364    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-var-lib-cni-bin\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446440    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-netd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446460    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446474    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446481    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-netd\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446481    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-netns\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446460    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-cni-bin\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446488    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-systemd\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446522    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-systemd\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446542    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-etc-kubernetes\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446567    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-sysctl-d\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446567    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-registration-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446735    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446751    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-kubernetes\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446616    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-etc-kubernetes\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446765    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-lib-modules\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446782    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-conf-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446785    4095 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446796    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-log-socket\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446812    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446829    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446837    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-cni-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446616    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-registration-dir\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446797    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-etc-kubernetes\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446844    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-log-socket\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446862    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-netns\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446851    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-multus-conf-dir\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446883    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-lib-modules\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446890    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/246aad42-5862-4e6e-af04-a7a9847f4ab5-host\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446890    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-netns\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446889    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-tuning-conf-dir\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446891    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-run-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446915    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/246aad42-5862-4e6e-af04-a7a9847f4ab5-host\") pod \"node-ca-ld2ml\" (UID: \"246aad42-5862-4e6e-af04-a7a9847f4ab5\") " pod="openshift-image-registry/node-ca-ld2ml"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.446955    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-k8s-cni-cncf-io\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.447003    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/04bb07a5-23a4-4095-a019-cb8a36e1bd63-host-run-k8s-cni-cncf-io\") pod \"multus-86859\" (UID: \"04bb07a5-23a4-4095-a019-cb8a36e1bd63\") " pod="openshift-multus/multus-86859"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.447008    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.447029    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-cnibin\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.447053    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-host\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.447061    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c76d9907-a77b-455f-9437-660fd451b40b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hpzwg\" (UID: \"c76d9907-a77b-455f-9437-660fd451b40b\") " pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.447070    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/267252e3-49bf-4478-90fb-2660c06b2b62-cnibin\") pod \"multus-additional-cni-plugins-qtgbr\" (UID: \"267252e3-49bf-4478-90fb-2660c06b2b62\") " pod="openshift-multus/multus-additional-cni-plugins-qtgbr"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.447075    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.447111    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b6a342b-6f0b-4701-919e-d46c24861d27-host\") pod \"tuned-vw5xj\" (UID: \"0b6a342b-6f0b-4701-919e-d46c24861d27\") " pod="openshift-cluster-node-tuning-operator/tuned-vw5xj"
Nov 01 12:22:21.447814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.447126    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d19a543b-1191-4040-b996-15e710bbb1bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-nkfgx\" (UID: \"d19a543b-1191-4040-b996-15e710bbb1bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-nkfgx"
Nov 01 12:22:21.449041 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.449027    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/752a2f8b-cf54-4bf3-bd4d-39ba739cb627-metrics-certs\") pod \"network-metrics-daemon-dlcrd\" (UID: \"752a2f8b-cf54-4bf3-bd4d-39ba739cb627\") " pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:21.544212 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.544169    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dlcrd"
Nov 01 12:22:21.904552 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.904521    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dlcrd"]
Nov 01 12:22:21.908092 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:21.908062    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752a2f8b_cf54_4bf3_bd4d_39ba739cb627.slice/crio-46cabe47f386a1df5bed476305ae7f34e35fc129c02abf3b0d4cfe4797289730 WatchSource:0}: Error finding container 46cabe47f386a1df5bed476305ae7f34e35fc129c02abf3b0d4cfe4797289730: Status 404 returned error can't find the container with id 46cabe47f386a1df5bed476305ae7f34e35fc129c02abf3b0d4cfe4797289730
Nov 01 12:22:21.910057 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:21.910040    4095 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:22:22.216687 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:22.216601    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dlcrd" event={"ID":"752a2f8b-cf54-4bf3-bd4d-39ba739cb627","Type":"ContainerStarted","Data":"46cabe47f386a1df5bed476305ae7f34e35fc129c02abf3b0d4cfe4797289730"}
Nov 01 12:22:23.194448 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:23.194411    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:23.717632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:23.717434    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:24.222307 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:24.222267    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dlcrd" event={"ID":"752a2f8b-cf54-4bf3-bd4d-39ba739cb627","Type":"ContainerStarted","Data":"3ab2fcbf23609870e0bf148f52d24b19b2a5d84a3d26ab7c05c5aa4ae6412558"}
Nov 01 12:22:24.222613 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:24.222321    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dlcrd" event={"ID":"752a2f8b-cf54-4bf3-bd4d-39ba739cb627","Type":"ContainerStarted","Data":"02592240e31e52d1a1a465b706a61a161cbaba7c792044c6b4fa519a9fd67e8c"}
Nov 01 12:22:28.927728 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:28.927696    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:28.928241 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:28.927893    4095 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 01 12:22:30.721673 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.721649    4095 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-140-168.us-west-2.compute.internal" event="NodeReady"
Nov 01 12:22:30.828890 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.828861    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-lx9tr"]
Nov 01 12:22:30.832447 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.832427    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6nfpg"]
Nov 01 12:22:30.832584 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.832568    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:30.834112 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.834088    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rxlpj"]
Nov 01 12:22:30.834231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.834130    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6nfpg"
Nov 01 12:22:30.835763 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.835747    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:30.882355 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.882328    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 01 12:22:30.882514 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.882506    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lt5kb\""
Nov 01 12:22:30.882602 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.882585    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 01 12:22:30.883287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.883274    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 01 12:22:30.883522 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.883509    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 01 12:22:30.883522 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.883517    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-86gmg\""
Nov 01 12:22:30.883599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.883529    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 01 12:22:30.886742 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.886720    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 01 12:22:30.887624 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.887605    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6nfpg"]
Nov 01 12:22:30.890554 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.890524    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lx9tr"]
Nov 01 12:22:30.950770 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.950740    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rxlpj"]
Nov 01 12:22:30.985355 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.985276    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 01 12:22:30.985520 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.985495    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-thbc5\""
Nov 01 12:22:30.985582 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.985543    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 01 12:22:30.985582 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:30.985552    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 01 12:22:31.000143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000117    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77246d24-a4e4-40e4-926f-778310f8f5a5-config-volume\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.000258 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000150    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghnx\" (UniqueName: \"kubernetes.io/projected/77246d24-a4e4-40e4-926f-778310f8f5a5-kube-api-access-jghnx\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.000258 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000188    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77246d24-a4e4-40e4-926f-778310f8f5a5-metrics-tls\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.000331 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000284    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fc58cf-a841-4b09-9b51-382f8ed5c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.000331 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000307    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53fc58cf-a841-4b09-9b51-382f8ed5c97e-crio-socket\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.000331 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000324    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcx95\" (UniqueName: \"kubernetes.io/projected/53fc58cf-a841-4b09-9b51-382f8ed5c97e-kube-api-access-pcx95\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.000452 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000342    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77246d24-a4e4-40e4-926f-778310f8f5a5-tmp-dir\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.000452 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000357    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qjzgj\" (UniqueName: \"kubernetes.io/projected/c330a645-238e-4bb8-9b6a-91ff333200b1-kube-api-access-qjzgj\") pod \"ingress-canary-6nfpg\" (UID: \"c330a645-238e-4bb8-9b6a-91ff333200b1\") " pod="openshift-ingress-canary/ingress-canary-6nfpg"
Nov 01 12:22:31.000517 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000461    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fc58cf-a841-4b09-9b51-382f8ed5c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.000551 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000538    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c330a645-238e-4bb8-9b6a-91ff333200b1-cert\") pod \"ingress-canary-6nfpg\" (UID: \"c330a645-238e-4bb8-9b6a-91ff333200b1\") " pod="openshift-ingress-canary/ingress-canary-6nfpg"
Nov 01 12:22:31.000580 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.000554    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53fc58cf-a841-4b09-9b51-382f8ed5c97e-data-volume\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.101790 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.101765    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77246d24-a4e4-40e4-926f-778310f8f5a5-config-volume\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.101790 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.101794    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghnx\" (UniqueName: \"kubernetes.io/projected/77246d24-a4e4-40e4-926f-778310f8f5a5-kube-api-access-jghnx\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.102013 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.101816    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77246d24-a4e4-40e4-926f-778310f8f5a5-metrics-tls\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.102074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102006    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fc58cf-a841-4b09-9b51-382f8ed5c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.102074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102040    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53fc58cf-a841-4b09-9b51-382f8ed5c97e-crio-socket\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.102074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102060    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcx95\" (UniqueName: \"kubernetes.io/projected/53fc58cf-a841-4b09-9b51-382f8ed5c97e-kube-api-access-pcx95\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.102231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102081    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77246d24-a4e4-40e4-926f-778310f8f5a5-tmp-dir\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.102231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102105    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qjzgj\" (UniqueName: \"kubernetes.io/projected/c330a645-238e-4bb8-9b6a-91ff333200b1-kube-api-access-qjzgj\") pod \"ingress-canary-6nfpg\" (UID: \"c330a645-238e-4bb8-9b6a-91ff333200b1\") " pod="openshift-ingress-canary/ingress-canary-6nfpg"
Nov 01 12:22:31.102231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102134    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fc58cf-a841-4b09-9b51-382f8ed5c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.102231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102162    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c330a645-238e-4bb8-9b6a-91ff333200b1-cert\") pod \"ingress-canary-6nfpg\" (UID: \"c330a645-238e-4bb8-9b6a-91ff333200b1\") " pod="openshift-ingress-canary/ingress-canary-6nfpg"
Nov 01 12:22:31.102231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102184    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53fc58cf-a841-4b09-9b51-382f8ed5c97e-data-volume\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.102231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102190    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53fc58cf-a841-4b09-9b51-382f8ed5c97e-crio-socket\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.102518 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102388    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77246d24-a4e4-40e4-926f-778310f8f5a5-tmp-dir\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.102655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102637    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53fc58cf-a841-4b09-9b51-382f8ed5c97e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.102720 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102661    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/77246d24-a4e4-40e4-926f-778310f8f5a5-config-volume\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.102720 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.102700    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53fc58cf-a841-4b09-9b51-382f8ed5c97e-data-volume\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.105795 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.105771    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53fc58cf-a841-4b09-9b51-382f8ed5c97e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.105876 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.105772    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/77246d24-a4e4-40e4-926f-778310f8f5a5-metrics-tls\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.105876 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.105815    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c330a645-238e-4bb8-9b6a-91ff333200b1-cert\") pod \"ingress-canary-6nfpg\" (UID: \"c330a645-238e-4bb8-9b6a-91ff333200b1\") " pod="openshift-ingress-canary/ingress-canary-6nfpg"
Nov 01 12:22:31.158744 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.158055    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghnx\" (UniqueName: \"kubernetes.io/projected/77246d24-a4e4-40e4-926f-778310f8f5a5-kube-api-access-jghnx\") pod \"dns-default-lx9tr\" (UID: \"77246d24-a4e4-40e4-926f-778310f8f5a5\") " pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.159775 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.159081    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qjzgj\" (UniqueName: \"kubernetes.io/projected/c330a645-238e-4bb8-9b6a-91ff333200b1-kube-api-access-qjzgj\") pod \"ingress-canary-6nfpg\" (UID: \"c330a645-238e-4bb8-9b6a-91ff333200b1\") " pod="openshift-ingress-canary/ingress-canary-6nfpg"
Nov 01 12:22:31.160784 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.160764    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcx95\" (UniqueName: \"kubernetes.io/projected/53fc58cf-a841-4b09-9b51-382f8ed5c97e-kube-api-access-pcx95\") pod \"insights-runtime-extractor-rxlpj\" (UID: \"53fc58cf-a841-4b09-9b51-382f8ed5c97e\") " pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.445742 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.445707    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:31.450049 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.450032    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6nfpg"
Nov 01 12:22:31.456287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.456232    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rxlpj"
Nov 01 12:22:31.511046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.510992    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-nsn56"
Nov 01 12:22:31.809190 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.809119    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6nfpg"]
Nov 01 12:22:31.855708 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.855634    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-lx9tr"]
Nov 01 12:22:31.857806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:31.857784    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77246d24_a4e4_40e4_926f_778310f8f5a5.slice/crio-a455c4da8b84795f801c7c0acc6df22a101888ffdae14949448fdd48b88590b2 WatchSource:0}: Error finding container a455c4da8b84795f801c7c0acc6df22a101888ffdae14949448fdd48b88590b2: Status 404 returned error can't find the container with id a455c4da8b84795f801c7c0acc6df22a101888ffdae14949448fdd48b88590b2
Nov 01 12:22:31.902169 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:31.902142    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rxlpj"]
Nov 01 12:22:31.904371 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:31.904350    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fc58cf_a841_4b09_9b51_382f8ed5c97e.slice/crio-b3cfd3b626356484d78dc28f489ecdc4695e64e7596bebfe0ccfbb7f9559efbb WatchSource:0}: Error finding container b3cfd3b626356484d78dc28f489ecdc4695e64e7596bebfe0ccfbb7f9559efbb: Status 404 returned error can't find the container with id b3cfd3b626356484d78dc28f489ecdc4695e64e7596bebfe0ccfbb7f9559efbb
Nov 01 12:22:32.242147 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:32.242114    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxlpj" event={"ID":"53fc58cf-a841-4b09-9b51-382f8ed5c97e","Type":"ContainerStarted","Data":"21e28e977be1064ea45befb68c992c8e491687bbc57ffbd8525135e81c3acff7"}
Nov 01 12:22:32.242307 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:32.242178    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxlpj" event={"ID":"53fc58cf-a841-4b09-9b51-382f8ed5c97e","Type":"ContainerStarted","Data":"b3cfd3b626356484d78dc28f489ecdc4695e64e7596bebfe0ccfbb7f9559efbb"}
Nov 01 12:22:32.243317 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:32.243293    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6nfpg" event={"ID":"c330a645-238e-4bb8-9b6a-91ff333200b1","Type":"ContainerStarted","Data":"731819486cb24e602160b46dd3d31c370db95017bd3d742135a8e0c99084a7f9"}
Nov 01 12:22:32.244340 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:32.244319    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lx9tr" event={"ID":"77246d24-a4e4-40e4-926f-778310f8f5a5","Type":"ContainerStarted","Data":"a455c4da8b84795f801c7c0acc6df22a101888ffdae14949448fdd48b88590b2"}
Nov 01 12:22:34.252271 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:34.252228    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxlpj" event={"ID":"53fc58cf-a841-4b09-9b51-382f8ed5c97e","Type":"ContainerStarted","Data":"189543e108d0fe893f0ac1a91044d9324437fbb570b8e403245527c024cb042d"}
Nov 01 12:22:34.254225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:34.254193    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lx9tr" event={"ID":"77246d24-a4e4-40e4-926f-778310f8f5a5","Type":"ContainerStarted","Data":"195d70f7506f83daa6b6d12c38f06126e4bc28c28b297fd3cf16fb03ddaf1dd9"}
Nov 01 12:22:34.254360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:34.254248    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-lx9tr" event={"ID":"77246d24-a4e4-40e4-926f-778310f8f5a5","Type":"ContainerStarted","Data":"3f487d7b883dfb9fbc52453398629ac1108c8036597f5e6a7cbe6b8bc8a000e4"}
Nov 01 12:22:34.254360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:34.254296    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:36.261541 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.261500    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxlpj" event={"ID":"53fc58cf-a841-4b09-9b51-382f8ed5c97e","Type":"ContainerStarted","Data":"5ca61eab0631791448bbab43b1082cd949b263565c6c3aee520c088ad6e5d1fe"}
Nov 01 12:22:36.370414 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.370355    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rxlpj" podStartSLOduration=3.091909776 podStartE2EDuration="6.370339222s" totalImagesPullingTime="3.278429446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 12:22:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:36.369843272 +0000 UTC m=+16.287638939" watchObservedRunningTime="2026-11-01 12:22:36.370339222 +0000 UTC m=+16.288134889"
Nov 01 12:22:36.370663 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.370636    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-lx9tr" podStartSLOduration=4.192827477 podStartE2EDuration="6.370629657s" totalImagesPullingTime="2.17780218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:34.363223869 +0000 UTC m=+14.281019534" watchObservedRunningTime="2026-11-01 12:22:36.370629657 +0000 UTC m=+16.288425326"
Nov 01 12:22:36.540918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.540833    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"]
Nov 01 12:22:36.543896 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.543875    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"
Nov 01 12:22:36.593597 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.593565    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 01 12:22:36.593746 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.593635    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-sksk5\""
Nov 01 12:22:36.597364 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.597338    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"]
Nov 01 12:22:36.639824 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.639791    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2de24e27-6407-4189-9fda-c8468c9bef28-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2f6kw\" (UID: \"2de24e27-6407-4189-9fda-c8468c9bef28\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"
Nov 01 12:22:36.743199 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.740435    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2de24e27-6407-4189-9fda-c8468c9bef28-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2f6kw\" (UID: \"2de24e27-6407-4189-9fda-c8468c9bef28\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"
Nov 01 12:22:36.743950 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.743924    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/2de24e27-6407-4189-9fda-c8468c9bef28-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-2f6kw\" (UID: \"2de24e27-6407-4189-9fda-c8468c9bef28\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"
Nov 01 12:22:36.854020 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:36.853919    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"
Nov 01 12:22:37.211391 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:37.211360    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"]
Nov 01 12:22:37.214573 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:37.214542    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2de24e27_6407_4189_9fda_c8468c9bef28.slice/crio-caac42e4b217ab1ed1888fb7e49f8e6b12a6be7196a7c8e1fbb777a5782e09d6 WatchSource:0}: Error finding container caac42e4b217ab1ed1888fb7e49f8e6b12a6be7196a7c8e1fbb777a5782e09d6: Status 404 returned error can't find the container with id caac42e4b217ab1ed1888fb7e49f8e6b12a6be7196a7c8e1fbb777a5782e09d6
Nov 01 12:22:37.266466 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:37.266432    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw" event={"ID":"2de24e27-6407-4189-9fda-c8468c9bef28","Type":"ContainerStarted","Data":"caac42e4b217ab1ed1888fb7e49f8e6b12a6be7196a7c8e1fbb777a5782e09d6"}
Nov 01 12:22:38.270268 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:38.270237    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6nfpg" event={"ID":"c330a645-238e-4bb8-9b6a-91ff333200b1","Type":"ContainerStarted","Data":"87cc38be0a574825c4d3ecab3e9898078fb3838a0332915662663099efd2b5e4"}
Nov 01 12:22:38.378762 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:38.378711    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6nfpg" podStartSLOduration=2.3827930889999998 podStartE2EDuration="8.378697453s" totalImagesPullingTime="5.995904364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:38.378082213 +0000 UTC m=+18.295877878" watchObservedRunningTime="2026-11-01 12:22:38.378697453 +0000 UTC m=+18.296493113"
Nov 01 12:22:39.274660 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:39.274572    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw" event={"ID":"2de24e27-6407-4189-9fda-c8468c9bef28","Type":"ContainerStarted","Data":"c8e30feca97545189120cbb099a464fe604adc84c0fc35b3cc26bde5baf7ab82"}
Nov 01 12:22:39.382441 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:39.382392    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw" podStartSLOduration=1.642930651 podStartE2EDuration="3.382378371s" totalImagesPullingTime="1.73944772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:39.381784109 +0000 UTC m=+19.299579776" watchObservedRunningTime="2026-11-01 12:22:39.382378371 +0000 UTC m=+19.300174034"
Nov 01 12:22:40.277550 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:40.277524    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"
Nov 01 12:22:40.282075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:40.282052    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-2f6kw"
Nov 01 12:22:41.117183 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.117144    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"]
Nov 01 12:22:41.120415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.120390    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.166635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.166602    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a6cb753-afaf-4be7-b341-cb0da3518da0-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.166743 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.166638    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxrf6\" (UniqueName: \"kubernetes.io/projected/3a6cb753-afaf-4be7-b341-cb0da3518da0-kube-api-access-rxrf6\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.166743 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.166728    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a6cb753-afaf-4be7-b341-cb0da3518da0-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.166811 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.166756    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a6cb753-afaf-4be7-b341-cb0da3518da0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.218260 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.218225    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Nov 01 12:22:41.218260 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.218258    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 01 12:22:41.218512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.218316    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-62ck5\""
Nov 01 12:22:41.218512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.218368    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 01 12:22:41.218512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.218427    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Nov 01 12:22:41.218512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.218491    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 01 12:22:41.266267 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.266236    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"]
Nov 01 12:22:41.267908 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.267893    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a6cb753-afaf-4be7-b341-cb0da3518da0-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.267956 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.267926    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a6cb753-afaf-4be7-b341-cb0da3518da0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.268010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.267958    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a6cb753-afaf-4be7-b341-cb0da3518da0-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.268010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.267990    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxrf6\" (UniqueName: \"kubernetes.io/projected/3a6cb753-afaf-4be7-b341-cb0da3518da0-kube-api-access-rxrf6\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.268635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.268609    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a6cb753-afaf-4be7-b341-cb0da3518da0-metrics-client-ca\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.270291 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.270270    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a6cb753-afaf-4be7-b341-cb0da3518da0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.270460 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.270441    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a6cb753-afaf-4be7-b341-cb0da3518da0-prometheus-operator-tls\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.321338 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.321313    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxrf6\" (UniqueName: \"kubernetes.io/projected/3a6cb753-afaf-4be7-b341-cb0da3518da0-kube-api-access-rxrf6\") pod \"prometheus-operator-9cd89b74c-8tt9c\" (UID: \"3a6cb753-afaf-4be7-b341-cb0da3518da0\") " pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.433497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.433393    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"
Nov 01 12:22:41.791398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:41.791329    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c"]
Nov 01 12:22:41.826253 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:41.826225    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6cb753_afaf_4be7_b341_cb0da3518da0.slice/crio-2648c92627edd945240775af7976063720c1b9825e22f73c63a2c2d280bbb5a9 WatchSource:0}: Error finding container 2648c92627edd945240775af7976063720c1b9825e22f73c63a2c2d280bbb5a9: Status 404 returned error can't find the container with id 2648c92627edd945240775af7976063720c1b9825e22f73c63a2c2d280bbb5a9
Nov 01 12:22:42.276430 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.276395    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-msvms"]
Nov 01 12:22:42.280712 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.280685    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:22:42.284846 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.284819    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c" event={"ID":"3a6cb753-afaf-4be7-b341-cb0da3518da0","Type":"ContainerStarted","Data":"2648c92627edd945240775af7976063720c1b9825e22f73c63a2c2d280bbb5a9"}
Nov 01 12:22:42.334139 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.334069    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 01 12:22:42.334542 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.334225    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-ttq55\""
Nov 01 12:22:42.376402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.376377    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfcrt\" (UniqueName: \"kubernetes.io/projected/1201afef-a55a-4f12-92b1-6afc127c3c88-kube-api-access-qfcrt\") pod \"downloads-6dd789c8b7-msvms\" (UID: \"1201afef-a55a-4f12-92b1-6afc127c3c88\") " pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:22:42.376509 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.376427    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1201afef-a55a-4f12-92b1-6afc127c3c88-tmp\") pod \"downloads-6dd789c8b7-msvms\" (UID: \"1201afef-a55a-4f12-92b1-6afc127c3c88\") " pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:22:42.381745 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.381729    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 01 12:22:42.382419 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.382399    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-msvms"]
Nov 01 12:22:42.477717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.477680    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1201afef-a55a-4f12-92b1-6afc127c3c88-tmp\") pod \"downloads-6dd789c8b7-msvms\" (UID: \"1201afef-a55a-4f12-92b1-6afc127c3c88\") " pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:22:42.477853 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.477735    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfcrt\" (UniqueName: \"kubernetes.io/projected/1201afef-a55a-4f12-92b1-6afc127c3c88-kube-api-access-qfcrt\") pod \"downloads-6dd789c8b7-msvms\" (UID: \"1201afef-a55a-4f12-92b1-6afc127c3c88\") " pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:22:42.478168 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.478151    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1201afef-a55a-4f12-92b1-6afc127c3c88-tmp\") pod \"downloads-6dd789c8b7-msvms\" (UID: \"1201afef-a55a-4f12-92b1-6afc127c3c88\") " pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:22:42.531598 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.531533    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfcrt\" (UniqueName: \"kubernetes.io/projected/1201afef-a55a-4f12-92b1-6afc127c3c88-kube-api-access-qfcrt\") pod \"downloads-6dd789c8b7-msvms\" (UID: \"1201afef-a55a-4f12-92b1-6afc127c3c88\") " pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:22:42.592444 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:42.592391    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:22:43.046501 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:43.046469    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1201afef_a55a_4f12_92b1_6afc127c3c88.slice/crio-f539b483712fe27a567813730d1e609c0213b3763442e9b501cef7ba325e0e75 WatchSource:0}: Error finding container f539b483712fe27a567813730d1e609c0213b3763442e9b501cef7ba325e0e75: Status 404 returned error can't find the container with id f539b483712fe27a567813730d1e609c0213b3763442e9b501cef7ba325e0e75
Nov 01 12:22:43.048286 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:43.048271    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-msvms"]
Nov 01 12:22:43.290209 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:43.290176    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-msvms" event={"ID":"1201afef-a55a-4f12-92b1-6afc127c3c88","Type":"ContainerStarted","Data":"f539b483712fe27a567813730d1e609c0213b3763442e9b501cef7ba325e0e75"}
Nov 01 12:22:44.259985 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:44.259940    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-lx9tr"
Nov 01 12:22:44.295824 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:44.295794    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c" event={"ID":"3a6cb753-afaf-4be7-b341-cb0da3518da0","Type":"ContainerStarted","Data":"3ef47dac5778a2e26c065b6a13569a375eb7a1262aedaade2e5d6c9f8bc8fbff"}
Nov 01 12:22:44.295986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:44.295842    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c" event={"ID":"3a6cb753-afaf-4be7-b341-cb0da3518da0","Type":"ContainerStarted","Data":"9338d335513d55226b60eaa5e39c1c5f6eaece0f9d72864e8da90aff0c9e29fb"}
Nov 01 12:22:44.481275 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:44.481220    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-9cd89b74c-8tt9c" podStartSLOduration=1.571413024 podStartE2EDuration="3.481206512s" totalImagesPullingTime="1.909793488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:44.480668706 +0000 UTC m=+24.398464374" watchObservedRunningTime="2026-11-01 12:22:44.481206512 +0000 UTC m=+24.399002226"
Nov 01 12:22:46.643024 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:46.642998    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hpzwg"
Nov 01 12:22:47.633890 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.633856    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"]
Nov 01 12:22:47.637607 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.637586    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.681118 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.681086    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-hj2v5"]
Nov 01 12:22:47.684361 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.684341    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.688962 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.688944    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Nov 01 12:22:47.720622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.720592    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3076378-1092-4108-90af-c56ce923a846-node-exporter-wtmp\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.720622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.720626    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3076378-1092-4108-90af-c56ce923a846-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.720825 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.720658    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.720825 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.720810    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.720906 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.720855    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.720906 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.720889    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3076378-1092-4108-90af-c56ce923a846-node-exporter-accelerators-collector-config\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.721030 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.720979    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7b5a5120-e40c-410c-9f68-93571e0467a6-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.721087 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.721037    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3076378-1092-4108-90af-c56ce923a846-root\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.721137 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.721094    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kb2q\" (UniqueName: \"kubernetes.io/projected/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-api-access-7kb2q\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.721137 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.721116    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3076378-1092-4108-90af-c56ce923a846-metrics-client-ca\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.721214 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.721144    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3076378-1092-4108-90af-c56ce923a846-node-exporter-tls\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.721265 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.721237    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3076378-1092-4108-90af-c56ce923a846-sys\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.721316 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.721264    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b5a5120-e40c-410c-9f68-93571e0467a6-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.721388 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.721363    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3076378-1092-4108-90af-c56ce923a846-node-exporter-textfile\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.721515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.721402    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfjh6\" (UniqueName: \"kubernetes.io/projected/d3076378-1092-4108-90af-c56ce923a846-kube-api-access-jfjh6\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.728079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.728056    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-gjx8v\""
Nov 01 12:22:47.736294 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.736275    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Nov 01 12:22:47.736461 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.736358    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Nov 01 12:22:47.740657 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.739372    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"]
Nov 01 12:22:47.793461 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.793428    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 01 12:22:47.793620 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.793538    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 01 12:22:47.793620 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.793543    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-ttg6g\""
Nov 01 12:22:47.793620 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.793613    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 01 12:22:47.822570 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822539    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.822714 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822586    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.822714 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822622    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.822714 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822652    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3076378-1092-4108-90af-c56ce923a846-node-exporter-accelerators-collector-config\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.822714 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822680    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7b5a5120-e40c-410c-9f68-93571e0467a6-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.822714 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822709    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3076378-1092-4108-90af-c56ce923a846-root\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.822989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822744    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kb2q\" (UniqueName: \"kubernetes.io/projected/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-api-access-7kb2q\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.822989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822770    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3076378-1092-4108-90af-c56ce923a846-metrics-client-ca\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.822989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822796    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3076378-1092-4108-90af-c56ce923a846-node-exporter-tls\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.822989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822825    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3076378-1092-4108-90af-c56ce923a846-sys\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.822989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822847    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b5a5120-e40c-410c-9f68-93571e0467a6-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.822989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822872    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3076378-1092-4108-90af-c56ce923a846-node-exporter-textfile\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.822989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822895    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfjh6\" (UniqueName: \"kubernetes.io/projected/d3076378-1092-4108-90af-c56ce923a846-kube-api-access-jfjh6\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.822989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822928    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3076378-1092-4108-90af-c56ce923a846-node-exporter-wtmp\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.822989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.822954    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3076378-1092-4108-90af-c56ce923a846-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.823456 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.823197    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d3076378-1092-4108-90af-c56ce923a846-root\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.823456 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.823415    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.823557 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.823512    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/7b5a5120-e40c-410c-9f68-93571e0467a6-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.823700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.823677    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d3076378-1092-4108-90af-c56ce923a846-node-exporter-textfile\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.823769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.823725    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d3076378-1092-4108-90af-c56ce923a846-sys\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.824074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.824049    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d3076378-1092-4108-90af-c56ce923a846-node-exporter-wtmp\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.824074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.824033    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d3076378-1092-4108-90af-c56ce923a846-metrics-client-ca\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.824458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.824431    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b5a5120-e40c-410c-9f68-93571e0467a6-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.825074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.825048    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d3076378-1092-4108-90af-c56ce923a846-node-exporter-accelerators-collector-config\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.825841 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.825819    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d3076378-1092-4108-90af-c56ce923a846-node-exporter-tls\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.826143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.826125    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d3076378-1092-4108-90af-c56ce923a846-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.826702 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.826681    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.827182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.827163    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.877896 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.877867    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kb2q\" (UniqueName: \"kubernetes.io/projected/7b5a5120-e40c-410c-9f68-93571e0467a6-kube-api-access-7kb2q\") pod \"kube-state-metrics-5dc8d7d89b-jkjf8\" (UID: \"7b5a5120-e40c-410c-9f68-93571e0467a6\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:47.878250 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.878230    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfjh6\" (UniqueName: \"kubernetes.io/projected/d3076378-1092-4108-90af-c56ce923a846-kube-api-access-jfjh6\") pod \"node-exporter-hj2v5\" (UID: \"d3076378-1092-4108-90af-c56ce923a846\") " pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:47.955218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:47.955134    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"
Nov 01 12:22:48.002365 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:48.002132    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-hj2v5"
Nov 01 12:22:48.019431 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:48.019392    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3076378_1092_4108_90af_c56ce923a846.slice/crio-deec8175184ab8a49bd44dafd07a32053661799451c446b145e9b8639f7d256e WatchSource:0}: Error finding container deec8175184ab8a49bd44dafd07a32053661799451c446b145e9b8639f7d256e: Status 404 returned error can't find the container with id deec8175184ab8a49bd44dafd07a32053661799451c446b145e9b8639f7d256e
Nov 01 12:22:48.311473 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:48.311394    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hj2v5" event={"ID":"d3076378-1092-4108-90af-c56ce923a846","Type":"ContainerStarted","Data":"deec8175184ab8a49bd44dafd07a32053661799451c446b145e9b8639f7d256e"}
Nov 01 12:22:48.366501 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:48.366470    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8"]
Nov 01 12:22:48.369601 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:48.369576    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b5a5120_e40c_410c_9f68_93571e0467a6.slice/crio-628dcc325b9dcb85d0eec194cb0a932d22bf916300158eeffd57064710f4efa2 WatchSource:0}: Error finding container 628dcc325b9dcb85d0eec194cb0a932d22bf916300158eeffd57064710f4efa2: Status 404 returned error can't find the container with id 628dcc325b9dcb85d0eec194cb0a932d22bf916300158eeffd57064710f4efa2
Nov 01 12:22:49.123094 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:22:49.123056    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:22:49.317358 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.317321    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8" event={"ID":"7b5a5120-e40c-410c-9f68-93571e0467a6","Type":"ContainerStarted","Data":"628dcc325b9dcb85d0eec194cb0a932d22bf916300158eeffd57064710f4efa2"}
Nov 01 12:22:49.840722 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.840675    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 12:22:49.846609 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.845790    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.896656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.896625    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 12:22:49.940409 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940381    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940589 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940431    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940589 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940458    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940673 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940611    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940673 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940648    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-web-config\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940753 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940678    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frlv\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-kube-api-access-4frlv\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940753 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940706    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940753 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940735    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940852 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940766    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940852 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940808    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940852 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940830    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940949 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940857    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-config-volume\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.940949 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.940879    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-config-out\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:49.991186 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.991147    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 12:22:49.991350 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.991241    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 12:22:49.992046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.992028    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tcjvl\""
Nov 01 12:22:49.992229 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.992210    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 12:22:49.992287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.992217    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 12:22:49.992287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.992263    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 12:22:49.992361 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.992304    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 12:22:49.992361 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.992343    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 12:22:49.992422 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.992369    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 12:22:49.992472 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:49.992453    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 12:22:50.042020 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.041983    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042020 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042025    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042053    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-config-volume\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042086    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-config-out\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042130    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042173    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042200    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042238    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042262    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-web-config\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042290    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frlv\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-kube-api-access-4frlv\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042316    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042356    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.042731 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.042487    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.043096 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.043068    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.043202 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.043130    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.044588 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.044491    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.045007 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.044985    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.046421 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.046373    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-config-out\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.046524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.046473    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.046524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.046483    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.046881 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.046852    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-config-volume\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.047611 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.047593    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.047766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.047747    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.047880 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.047849    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-web-config\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.048014 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.047987    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.102449 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.102418    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frlv\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-kube-api-access-4frlv\") pod \"alertmanager-main-0\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.150391 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.150320    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"]
Nov 01 12:22:50.156356 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.156212    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.173009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.172963    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:22:50.207006 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.206901    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 01 12:22:50.207006 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.207003    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 01 12:22:50.207228 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.207097    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-2h65ebfen4qnu\""
Nov 01 12:22:50.207228 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.207171    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 01 12:22:50.210027 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.210005    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-n8xtw\""
Nov 01 12:22:50.215245 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.215179    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"]
Nov 01 12:22:50.245143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.245118    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-tmp\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.245253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.245162    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-grpc-tls\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.245253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.245225    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-tls\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.245366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.245258    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.245366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.245276    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-metrics-client-ca\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.245366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.245295    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.245366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.245329    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.245366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.245353    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.245626 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.245379    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89v6l\" (UniqueName: \"kubernetes.io/projected/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-kube-api-access-89v6l\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.307192 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.307158    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 01 12:22:50.307379 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.307279    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 01 12:22:50.323629 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.322904    4095 generic.go:356] "Generic (PLEG): container finished" podID="d3076378-1092-4108-90af-c56ce923a846" containerID="0e0892fa61e056bd5a29ad7c234f7607f5d4368412b1fc3639fab908504f0177" exitCode=0
Nov 01 12:22:50.323629 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.323011    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hj2v5" event={"ID":"d3076378-1092-4108-90af-c56ce923a846","Type":"ContainerDied","Data":"0e0892fa61e056bd5a29ad7c234f7607f5d4368412b1fc3639fab908504f0177"}
Nov 01 12:22:50.346525 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.346492    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.346672 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.346541    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.346672 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.346586    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89v6l\" (UniqueName: \"kubernetes.io/projected/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-kube-api-access-89v6l\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.346672 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.346624    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-tmp\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.346672 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.346650    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-grpc-tls\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.346911 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.346675    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-tls\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.346911 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.346749    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.346911 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.346775    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-metrics-client-ca\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.346911 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.346803    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.348706 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.348050    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-metrics-client-ca\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.348706 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.348349    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-tmp\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.349382 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.349352    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.349630 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.349592    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-tls\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.349740 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.349718    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.350165 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.350123    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.351659 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.351641    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-grpc-tls\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.353439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.353413    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.400702 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.400622    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89v6l\" (UniqueName: \"kubernetes.io/projected/9bff35e5-ff1f-4dfd-b05d-f87d37d9011e-kube-api-access-89v6l\") pod \"thanos-querier-7c896dfbdb-ldfhk\" (UID: \"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e\") " pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.484209 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.484176    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:50.600861 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.600833    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 12:22:50.604679 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:50.604651    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda18aad7_927f_4f7f_baca_07ff38303dd1.slice/crio-0408e4c3c7f9eab5efc847f78eb88d285de19c2de0ceca1889f2a9bafdd23377 WatchSource:0}: Error finding container 0408e4c3c7f9eab5efc847f78eb88d285de19c2de0ceca1889f2a9bafdd23377: Status 404 returned error can't find the container with id 0408e4c3c7f9eab5efc847f78eb88d285de19c2de0ceca1889f2a9bafdd23377
Nov 01 12:22:50.859143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:50.858862    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"]
Nov 01 12:22:50.861242 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:50.861207    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bff35e5_ff1f_4dfd_b05d_f87d37d9011e.slice/crio-bf32e60aa7766433b126647501c1d45873a771647f4243d985f4d2aa014836d6 WatchSource:0}: Error finding container bf32e60aa7766433b126647501c1d45873a771647f4243d985f4d2aa014836d6: Status 404 returned error can't find the container with id bf32e60aa7766433b126647501c1d45873a771647f4243d985f4d2aa014836d6
Nov 01 12:22:51.329869 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:51.329829    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hj2v5" event={"ID":"d3076378-1092-4108-90af-c56ce923a846","Type":"ContainerStarted","Data":"099340d593f814926bd1554740535d83eae5bd8bdc4a24f34e0e348ba8a00472"}
Nov 01 12:22:51.330260 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:51.329885    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-hj2v5" event={"ID":"d3076378-1092-4108-90af-c56ce923a846","Type":"ContainerStarted","Data":"9618302ec182e6079e2f64548e3b78a0719f261577a7255c6ef84d36ae8bc865"}
Nov 01 12:22:51.332808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:51.332678    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8" event={"ID":"7b5a5120-e40c-410c-9f68-93571e0467a6","Type":"ContainerStarted","Data":"6be9fff71afaf31031ff1e917fa24c447d7893fd0f60e1221d5ec08a46a50cb1"}
Nov 01 12:22:51.332808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:51.332740    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8" event={"ID":"7b5a5120-e40c-410c-9f68-93571e0467a6","Type":"ContainerStarted","Data":"203a177f6ab84133832f4599d927f96a70f13c9cc7514f8f230e3b2d49537234"}
Nov 01 12:22:51.332808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:51.332771    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8" event={"ID":"7b5a5120-e40c-410c-9f68-93571e0467a6","Type":"ContainerStarted","Data":"098a731726df4a1c156379e733c69de0f83f71a3ffb38f70b9970b69a81b443a"}
Nov 01 12:22:51.334244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:51.334215    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk" event={"ID":"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e","Type":"ContainerStarted","Data":"bf32e60aa7766433b126647501c1d45873a771647f4243d985f4d2aa014836d6"}
Nov 01 12:22:51.335924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:51.335897    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerStarted","Data":"0408e4c3c7f9eab5efc847f78eb88d285de19c2de0ceca1889f2a9bafdd23377"}
Nov 01 12:22:51.439419 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:51.439364    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-hj2v5" podStartSLOduration=2.7730139510000003 podStartE2EDuration="4.439350693s" totalImagesPullingTime="1.580159266s" totalInitContainerRuntime="86.177476ms" isStatefulPod=true podCreationTimestamp="2026-11-01 12:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:51.438615787 +0000 UTC m=+31.356411453" watchObservedRunningTime="2026-11-01 12:22:51.439350693 +0000 UTC m=+31.357146376"
Nov 01 12:22:51.548871 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:51.548813    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-jkjf8" podStartSLOduration=2.394200901 podStartE2EDuration="4.54879896s" totalImagesPullingTime="2.154598059s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:51.548053557 +0000 UTC m=+31.465849226" watchObservedRunningTime="2026-11-01 12:22:51.54879896 +0000 UTC m=+31.466594626"
Nov 01 12:22:52.450041 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.450007    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-6d87d6bbc7-df484"]
Nov 01 12:22:52.465286 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.465259    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.507733 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.507642    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d87d6bbc7-df484"]
Nov 01 12:22:52.516488 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.516459    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 01 12:22:52.516488 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.516484    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 01 12:22:52.516715 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.516494    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-9nf4k\""
Nov 01 12:22:52.516715 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.516524    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dpug319rcgs9u\""
Nov 01 12:22:52.516715 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.516549    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 01 12:22:52.516715 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.516559    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 01 12:22:52.566228 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.566191    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/016791b0-0300-4229-a88e-2446533ce831-client-ca-bundle\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.566372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.566270    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/016791b0-0300-4229-a88e-2446533ce831-metrics-server-audit-profiles\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.566372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.566328    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/016791b0-0300-4229-a88e-2446533ce831-secret-metrics-server-client-certs\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.566470 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.566368    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/016791b0-0300-4229-a88e-2446533ce831-secret-metrics-server-tls\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.566470 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.566423    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016791b0-0300-4229-a88e-2446533ce831-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.566564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.566477    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/016791b0-0300-4229-a88e-2446533ce831-audit-log\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.566564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.566514    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v8xc\" (UniqueName: \"kubernetes.io/projected/016791b0-0300-4229-a88e-2446533ce831-kube-api-access-7v8xc\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.667294 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.667265    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/016791b0-0300-4229-a88e-2446533ce831-audit-log\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.667477 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.667312    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v8xc\" (UniqueName: \"kubernetes.io/projected/016791b0-0300-4229-a88e-2446533ce831-kube-api-access-7v8xc\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.667477 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.667345    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/016791b0-0300-4229-a88e-2446533ce831-client-ca-bundle\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.667477 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.667369    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/016791b0-0300-4229-a88e-2446533ce831-metrics-server-audit-profiles\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.667477 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.667395    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/016791b0-0300-4229-a88e-2446533ce831-secret-metrics-server-client-certs\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.667477 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.667430    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/016791b0-0300-4229-a88e-2446533ce831-secret-metrics-server-tls\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.667477 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.667472    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016791b0-0300-4229-a88e-2446533ce831-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.667740 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.667712    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/016791b0-0300-4229-a88e-2446533ce831-audit-log\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.668603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.668583    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/016791b0-0300-4229-a88e-2446533ce831-metrics-server-audit-profiles\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.669286 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.669234    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/016791b0-0300-4229-a88e-2446533ce831-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.671270 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.671245    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/016791b0-0300-4229-a88e-2446533ce831-client-ca-bundle\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.671460 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.671441    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/016791b0-0300-4229-a88e-2446533ce831-secret-metrics-server-client-certs\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.671497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.671477    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/016791b0-0300-4229-a88e-2446533ce831-secret-metrics-server-tls\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.722361 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.722331    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v8xc\" (UniqueName: \"kubernetes.io/projected/016791b0-0300-4229-a88e-2446533ce831-kube-api-access-7v8xc\") pod \"metrics-server-6d87d6bbc7-df484\" (UID: \"016791b0-0300-4229-a88e-2446533ce831\") " pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:52.797941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:52.797902    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:22:53.197543 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.197510    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gnbj7"
Nov 01 12:22:53.254378 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.254347    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-6d87d6bbc7-df484"]
Nov 01 12:22:53.258827 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:53.258701    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016791b0_0300_4229_a88e_2446533ce831.slice/crio-6562aa0bf9eeddf8cd5e4127ebb58eb469a1c511a19482cf5c551ecac8b40c24 WatchSource:0}: Error finding container 6562aa0bf9eeddf8cd5e4127ebb58eb469a1c511a19482cf5c551ecac8b40c24: Status 404 returned error can't find the container with id 6562aa0bf9eeddf8cd5e4127ebb58eb469a1c511a19482cf5c551ecac8b40c24
Nov 01 12:22:53.354580 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.354538    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484" event={"ID":"016791b0-0300-4229-a88e-2446533ce831","Type":"ContainerStarted","Data":"6562aa0bf9eeddf8cd5e4127ebb58eb469a1c511a19482cf5c551ecac8b40c24"}
Nov 01 12:22:53.356512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.356479    4095 generic.go:356] "Generic (PLEG): container finished" podID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerID="cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf" exitCode=0
Nov 01 12:22:53.356658 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.356555    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerDied","Data":"cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf"}
Nov 01 12:22:53.737916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.737882    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6489db9f76-qs6gl"]
Nov 01 12:22:53.751544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.751516    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.796293 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.796260    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6489db9f76-qs6gl"]
Nov 01 12:22:53.801556 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.801529    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Nov 01 12:22:53.801692 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.801588    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Nov 01 12:22:53.801692 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.801631    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Nov 01 12:22:53.801692 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.801675    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Nov 01 12:22:53.801851 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.801690    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Nov 01 12:22:53.801851 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.801760    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-69snc\""
Nov 01 12:22:53.879084 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.879045    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8941ba26-4008-4b09-be55-56c225259ec5-tmp\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.879268 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.879098    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-oauth-serving-cert\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.879268 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.879200    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-oauth-config\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.879268 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.879262    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkgfx\" (UniqueName: \"kubernetes.io/projected/8941ba26-4008-4b09-be55-56c225259ec5-kube-api-access-nkgfx\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.879410 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.879369    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-console-config\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.879410 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.879400    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-service-ca\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.879484 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.879433    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-serving-cert\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.980436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.980359    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkgfx\" (UniqueName: \"kubernetes.io/projected/8941ba26-4008-4b09-be55-56c225259ec5-kube-api-access-nkgfx\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.980436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.980402    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-console-config\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.980436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.980420    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-service-ca\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.980755 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.980449    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-serving-cert\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.980755 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.980489    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8941ba26-4008-4b09-be55-56c225259ec5-tmp\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.980755 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.980519    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-oauth-serving-cert\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.980755 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.980552    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-oauth-config\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.981120 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.981094    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8941ba26-4008-4b09-be55-56c225259ec5-tmp\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.981229 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.981136    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-console-config\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.981284 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.981253    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-service-ca\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.981668 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.981644    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-oauth-serving-cert\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.983255 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.983234    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-serving-cert\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:53.983363 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:53.983236    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-oauth-config\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:54.037019 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:54.036990    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkgfx\" (UniqueName: \"kubernetes.io/projected/8941ba26-4008-4b09-be55-56c225259ec5-kube-api-access-nkgfx\") pod \"console-6489db9f76-qs6gl\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") " pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:54.076829 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:54.076797    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:22:54.362772 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:54.362747    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk" event={"ID":"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e","Type":"ContainerStarted","Data":"eb8a2cdd6f4094608e6fd85d17d993bf951af6dec6dd4473d8cf35b1d39e2a82"}
Nov 01 12:22:54.362869 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:54.362812    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk" event={"ID":"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e","Type":"ContainerStarted","Data":"0050182e9d8df5e43886f72fab991da2c3887aa50da9dbeed4020a2a539bb7b8"}
Nov 01 12:22:54.458519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:54.458485    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6489db9f76-qs6gl"]
Nov 01 12:22:54.461060 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:54.461033    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8941ba26_4008_4b09_be55_56c225259ec5.slice/crio-760e66a8b88d25743573ef8c65c5fea5d37ad45e76a5671e6b292dc9f31c9b10 WatchSource:0}: Error finding container 760e66a8b88d25743573ef8c65c5fea5d37ad45e76a5671e6b292dc9f31c9b10: Status 404 returned error can't find the container with id 760e66a8b88d25743573ef8c65c5fea5d37ad45e76a5671e6b292dc9f31c9b10
Nov 01 12:22:55.373217 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:55.373174    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk" event={"ID":"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e","Type":"ContainerStarted","Data":"b3c82f3263cd3a0b639e0cac3fbaaa6628f042373eb92f0dd834c5a888d5ef41"}
Nov 01 12:22:55.375506 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:55.375437    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6489db9f76-qs6gl" event={"ID":"8941ba26-4008-4b09-be55-56c225259ec5","Type":"ContainerStarted","Data":"760e66a8b88d25743573ef8c65c5fea5d37ad45e76a5671e6b292dc9f31c9b10"}
Nov 01 12:22:56.384935 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.384155    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk" event={"ID":"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e","Type":"ContainerStarted","Data":"5649f56c2cfb4329ddfcdbde3e1fc3ce6ec3f6f3124878b61813d4eeda2d1abd"}
Nov 01 12:22:56.386668 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.386639    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484" event={"ID":"016791b0-0300-4229-a88e-2446533ce831","Type":"ContainerStarted","Data":"91b17e81b504401f64b6f38cd311699bcc1738f71d9a65d62afa1670b72d6050"}
Nov 01 12:22:56.389988 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.389955    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerStarted","Data":"d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2"}
Nov 01 12:22:56.390055 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.390034    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerStarted","Data":"286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff"}
Nov 01 12:22:56.390116 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.390063    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerStarted","Data":"81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c"}
Nov 01 12:22:56.390116 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.390086    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerStarted","Data":"50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58"}
Nov 01 12:22:56.506990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.489633    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 12:22:56.506990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.493675    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.538555 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.536415    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484" podStartSLOduration=1.6590997299999999 podStartE2EDuration="4.536400151s" totalImagesPullingTime="2.877300421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:56.526238672 +0000 UTC m=+36.444034337" watchObservedRunningTime="2026-11-01 12:22:56.536400151 +0000 UTC m=+36.454195814"
Nov 01 12:22:56.546253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.545960    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 12:22:56.546253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.546086    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 12:22:56.546253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.546212    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 12:22:56.565191 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.565162    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 12:22:56.565391 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.565215    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 12:22:56.565534 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.565517    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 12:22:56.565650 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.565636    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 12:22:56.573865 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.573835    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 12:22:56.574013 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.573941    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e8109ldfulpk9\""
Nov 01 12:22:56.574108 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.574090    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-scdt8\""
Nov 01 12:22:56.574160 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.574152    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 12:22:56.574290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.574276    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 12:22:56.574357 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.574344    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 12:22:56.574429 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.574417    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609304    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609354    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609387    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609422    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwtvr\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-kube-api-access-nwtvr\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609448    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609477    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609502    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609539    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609559    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609580    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609607    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609640    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609677    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609702    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609726    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609761    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609783    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609820    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609855    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.610220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.609881    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.677136 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.677100    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 12:22:56.677306 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.677153    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 12:22:56.679213 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.679190    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 12:22:56.710484 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710453    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710650 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710494    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710650 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710524    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710650 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710559    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710650 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710585    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710650 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710625    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710676    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710715    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710752    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710793    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710822    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710851    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwtvr\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-kube-api-access-nwtvr\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710874    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.710902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710900    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.711351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.710923    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.711576 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.711551    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.711825 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.711799    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.712322 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.712297    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.713611    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.713651    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.713678    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.713711    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.713748    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.713741    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.714171    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714581 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.714371    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714581 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.714466    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.714790 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.714764    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.715450 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.715415    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.715768 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.715748    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.715856 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.715752    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.715919 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.715896    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.716449 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.716353    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config-out\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.716545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.716509    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.717129 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.717106    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.717384 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.717366    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.717779 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.717756    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-web-config\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.717952 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.717937    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.718403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.718370    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.767604 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.767535    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwtvr\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-kube-api-access-nwtvr\") pod \"prometheus-k8s-0\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:56.818392 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:56.818146    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:22:57.244671 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.244641    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 12:22:57.246769 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:22:57.246741    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aa78254_1892_4d5f_83f3_77abc7a5e39c.slice/crio-8b6a2a4793bee57bc8faf5bd135ed515869ddc71035b4db716710e15250676f1 WatchSource:0}: Error finding container 8b6a2a4793bee57bc8faf5bd135ed515869ddc71035b4db716710e15250676f1: Status 404 returned error can't find the container with id 8b6a2a4793bee57bc8faf5bd135ed515869ddc71035b4db716710e15250676f1
Nov 01 12:22:57.398683 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.398643    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerStarted","Data":"71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0"}
Nov 01 12:22:57.399192 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.398733    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerStarted","Data":"a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798"}
Nov 01 12:22:57.402423 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.402393    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk" event={"ID":"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e","Type":"ContainerStarted","Data":"2e3cae9e1bd42e99a6caab32f1e30a3a9bae88b5a4c8301c8b0c696dc0d86ab5"}
Nov 01 12:22:57.402633 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.402458    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk" event={"ID":"9bff35e5-ff1f-4dfd-b05d-f87d37d9011e","Type":"ContainerStarted","Data":"6d564d3aa10df9df8d1c3d14de53051cece89127ca17fa7d1239caa604a592ae"}
Nov 01 12:22:57.402633 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.402611    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:22:57.404678 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.404651    4095 generic.go:356] "Generic (PLEG): container finished" podID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerID="a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2" exitCode=0
Nov 01 12:22:57.405051 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.405024    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerDied","Data":"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"}
Nov 01 12:22:57.405154 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.405096    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerStarted","Data":"8b6a2a4793bee57bc8faf5bd135ed515869ddc71035b4db716710e15250676f1"}
Nov 01 12:22:57.515995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.515872    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.774460599 podStartE2EDuration="8.515852063s" totalImagesPullingTime="4.719367306s" totalInitContainerRuntime="22.024158ms" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:57.515200359 +0000 UTC m=+37.432996050" watchObservedRunningTime="2026-11-01 12:22:57.515852063 +0000 UTC m=+37.433647731"
Nov 01 12:22:57.746084 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:57.745528    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk" podStartSLOduration=2.6925721190000003 podStartE2EDuration="7.745510817s" totalImagesPullingTime="5.052938698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:57.745144485 +0000 UTC m=+37.662940165" watchObservedRunningTime="2026-11-01 12:22:57.745510817 +0000 UTC m=+37.663306485"
Nov 01 12:22:59.416590 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:59.416544    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6489db9f76-qs6gl" event={"ID":"8941ba26-4008-4b09-be55-56c225259ec5","Type":"ContainerStarted","Data":"c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918"}
Nov 01 12:22:59.527697 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:22:59.527633    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6489db9f76-qs6gl" podStartSLOduration=2.339294502 podStartE2EDuration="6.527609929s" totalImagesPullingTime="4.188315427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:22:59.527105578 +0000 UTC m=+39.444901244" watchObservedRunningTime="2026-11-01 12:22:59.527609929 +0000 UTC m=+39.445405595"
Nov 01 12:23:02.436655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:02.436627    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerStarted","Data":"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"}
Nov 01 12:23:02.436990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:02.436689    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerStarted","Data":"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"}
Nov 01 12:23:02.436990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:02.436716    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerStarted","Data":"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"}
Nov 01 12:23:02.436990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:02.436739    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerStarted","Data":"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"}
Nov 01 12:23:02.436990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:02.436760    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerStarted","Data":"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"}
Nov 01 12:23:03.417244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:03.417211    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7c896dfbdb-ldfhk"
Nov 01 12:23:03.446801 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:03.446767    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerStarted","Data":"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"}
Nov 01 12:23:03.448544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:03.448511    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-msvms" event={"ID":"1201afef-a55a-4f12-92b1-6afc127c3c88","Type":"ContainerStarted","Data":"c17bc0744d1cd4f006ef77a70d8a30f431594bdfce51b6d1c1471eb62d5f1a52"}
Nov 01 12:23:03.448759 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:03.448731    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:23:03.465229 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:03.465203    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-msvms"
Nov 01 12:23:03.698112 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:03.698001    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.233809424 podStartE2EDuration="7.697982408s" totalImagesPullingTime="4.441888575s" totalInitContainerRuntime="22.284409ms" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:23:03.696020766 +0000 UTC m=+43.613816432" watchObservedRunningTime="2026-11-01 12:23:03.697982408 +0000 UTC m=+43.615778072"
Nov 01 12:23:03.805746 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:03.805686    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-msvms" podStartSLOduration=2.209390264 podStartE2EDuration="21.805671274s" totalImagesPullingTime="19.59628101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:22:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:23:03.804208689 +0000 UTC m=+43.722004355" watchObservedRunningTime="2026-11-01 12:23:03.805671274 +0000 UTC m=+43.723466939"
Nov 01 12:23:04.077244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:04.077133    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:23:04.077244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:04.077226    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:23:04.083050 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:04.083021    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:23:04.459477 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:04.459444    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:23:06.818727 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:06.818685    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:12.798579 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:12.798549    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:23:12.799067 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:12.798598    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:23:15.639655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:15.639623    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:15.687576 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:15.687536    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:16.519711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:16.519677    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:17.774162 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:17.774123    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6489db9f76-qs6gl"]
Nov 01 12:23:19.111265 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:19.111224    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:23:32.807419 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:32.807389    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:23:32.811234 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:32.811211    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-6d87d6bbc7-df484"
Nov 01 12:23:41.139149 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.139114    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 12:23:41.139700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.139395    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="alertmanager" containerID="cri-o://50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58" gracePeriod=120
Nov 01 12:23:41.139700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.139471    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy-metric" containerID="cri-o://a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798" gracePeriod=120
Nov 01 12:23:41.139700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.139505    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy-web" containerID="cri-o://286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff" gracePeriod=120
Nov 01 12:23:41.139700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.139534    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="prom-label-proxy" containerID="cri-o://71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0" gracePeriod=120
Nov 01 12:23:41.139700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.139555    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="config-reloader" containerID="cri-o://81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c" gracePeriod=120
Nov 01 12:23:41.139700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.139623    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy" containerID="cri-o://d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2" gracePeriod=120
Nov 01 12:23:41.597708 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.597629    4095 generic.go:356] "Generic (PLEG): container finished" podID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerID="71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0" exitCode=0
Nov 01 12:23:41.597708 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.597655    4095 generic.go:356] "Generic (PLEG): container finished" podID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerID="d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2" exitCode=0
Nov 01 12:23:41.597708 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.597664    4095 generic.go:356] "Generic (PLEG): container finished" podID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerID="81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c" exitCode=0
Nov 01 12:23:41.597708 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.597671    4095 generic.go:356] "Generic (PLEG): container finished" podID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerID="50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58" exitCode=0
Nov 01 12:23:41.597931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.597701    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerDied","Data":"71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0"}
Nov 01 12:23:41.597931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.597774    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerDied","Data":"d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2"}
Nov 01 12:23:41.597931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.597804    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerDied","Data":"81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c"}
Nov 01 12:23:41.597931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:41.597829    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerDied","Data":"50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58"}
Nov 01 12:23:42.427498 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.427476    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.514451 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514384    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.514451 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514429    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-main-tls\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.514635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514454    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-config-out\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-config-out\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.514635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514474    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-trusted-ca-bundle\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.514635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514495    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-web-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-web-config\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.514635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514534    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-main-db\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.514635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514559    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-web\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.514635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514593    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-metrics-client-ca\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.514635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514631    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-kube-api-access-4frlv\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-kube-api-access-4frlv\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.515482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514658    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-config-volume\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-config-volume\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.515482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514694    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-tls-assets\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-tls-assets\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.515482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514764    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.515482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514796    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-cluster-tls-config\") pod \"da18aad7-927f-4f7f-baca-07ff38303dd1\" (UID: \"da18aad7-927f-4f7f-baca-07ff38303dd1\") "
Nov 01 12:23:42.515482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514882    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-main-db" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:23:42.515482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.514951    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-trusted-ca-bundle" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:42.515482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.515110    4095 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-main-db\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.515482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.515131    4095 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.515482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.515349    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-metrics-client-ca" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:42.517643 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.517597    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-main-tls" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:42.517769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.517644    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-config-volume" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:42.517870 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.517838    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-tls-assets" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:23:42.518401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.518364    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-web" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:42.518560 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.518524    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-metric" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:42.518638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.518604    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-config-out" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:23:42.518796 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.518778    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-kube-api-access-4frlv" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "kube-api-access-4frlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:23:42.519090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.519068    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:42.522224 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.521960    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-cluster-tls-config" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:42.528873 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.528851    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-web-config" pod "da18aad7-927f-4f7f-baca-07ff38303dd1" (UID: "da18aad7-927f-4f7f-baca-07ff38303dd1"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:42.604479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.604456    4095 generic.go:356] "Generic (PLEG): container finished" podID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerID="a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798" exitCode=0
Nov 01 12:23:42.604479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.604476    4095 generic.go:356] "Generic (PLEG): container finished" podID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerID="286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff" exitCode=0
Nov 01 12:23:42.604632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.604538    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.604632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.604540    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerDied","Data":"a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798"}
Nov 01 12:23:42.604632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.604620    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerDied","Data":"286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff"}
Nov 01 12:23:42.604764 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.604662    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"da18aad7-927f-4f7f-baca-07ff38303dd1","Type":"ContainerDied","Data":"0408e4c3c7f9eab5efc847f78eb88d285de19c2de0ceca1889f2a9bafdd23377"}
Nov 01 12:23:42.604764 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.604691    4095 scope.go:122] "RemoveContainer" containerID="71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0"
Nov 01 12:23:42.615378 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615362    4095 scope.go:122] "RemoveContainer" containerID="a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798"
Nov 01 12:23:42.615792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615774    4095 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.615862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615795    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da18aad7-927f-4f7f-baca-07ff38303dd1-metrics-client-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.615862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615805    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4frlv\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-kube-api-access-4frlv\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.615862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615814    4095 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-config-volume\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.615862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615824    4095 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/da18aad7-927f-4f7f-baca-07ff38303dd1-tls-assets\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.615862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615833    4095 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.615862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615841    4095 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-cluster-tls-config\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.615862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615850    4095 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.615862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615858    4095 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-secret-alertmanager-main-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.615862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615866    4095 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/da18aad7-927f-4f7f-baca-07ff38303dd1-config-out\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.616286 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.615875    4095 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/da18aad7-927f-4f7f-baca-07ff38303dd1-web-config\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:42.621914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.621895    4095 scope.go:122] "RemoveContainer" containerID="d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2"
Nov 01 12:23:42.629427 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.629407    4095 scope.go:122] "RemoveContainer" containerID="286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff"
Nov 01 12:23:42.636695 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.636614    4095 scope.go:122] "RemoveContainer" containerID="81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c"
Nov 01 12:23:42.645505 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.645485    4095 scope.go:122] "RemoveContainer" containerID="50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58"
Nov 01 12:23:42.652016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.652001    4095 scope.go:122] "RemoveContainer" containerID="cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf"
Nov 01 12:23:42.658654 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.658635    4095 scope.go:122] "RemoveContainer" containerID="71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0"
Nov 01 12:23:42.658944 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:42.658925    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0\": container with ID starting with 71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0 not found: ID does not exist" containerID="71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0"
Nov 01 12:23:42.659090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.658953    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0"} err="failed to get container status \"71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0\": rpc error: code = NotFound desc = could not find container \"71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0\": container with ID starting with 71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0 not found: ID does not exist"
Nov 01 12:23:42.659090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.659011    4095 scope.go:122] "RemoveContainer" containerID="a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798"
Nov 01 12:23:42.659255 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:42.659238    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798\": container with ID starting with a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798 not found: ID does not exist" containerID="a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798"
Nov 01 12:23:42.659294 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.659262    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798"} err="failed to get container status \"a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798\": rpc error: code = NotFound desc = could not find container \"a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798\": container with ID starting with a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798 not found: ID does not exist"
Nov 01 12:23:42.659294 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.659277    4095 scope.go:122] "RemoveContainer" containerID="d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2"
Nov 01 12:23:42.659515 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:42.659489    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2\": container with ID starting with d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2 not found: ID does not exist" containerID="d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2"
Nov 01 12:23:42.659590 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.659533    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2"} err="failed to get container status \"d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2\": rpc error: code = NotFound desc = could not find container \"d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2\": container with ID starting with d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2 not found: ID does not exist"
Nov 01 12:23:42.659590 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.659554    4095 scope.go:122] "RemoveContainer" containerID="286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff"
Nov 01 12:23:42.659787 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:42.659764    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff\": container with ID starting with 286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff not found: ID does not exist" containerID="286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff"
Nov 01 12:23:42.659833 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.659795    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff"} err="failed to get container status \"286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff\": rpc error: code = NotFound desc = could not find container \"286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff\": container with ID starting with 286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff not found: ID does not exist"
Nov 01 12:23:42.659833 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.659807    4095 scope.go:122] "RemoveContainer" containerID="81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c"
Nov 01 12:23:42.660063 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:42.660042    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c\": container with ID starting with 81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c not found: ID does not exist" containerID="81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c"
Nov 01 12:23:42.660122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.660069    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c"} err="failed to get container status \"81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c\": rpc error: code = NotFound desc = could not find container \"81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c\": container with ID starting with 81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c not found: ID does not exist"
Nov 01 12:23:42.660122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.660085    4095 scope.go:122] "RemoveContainer" containerID="50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58"
Nov 01 12:23:42.660336 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:42.660320    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58\": container with ID starting with 50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58 not found: ID does not exist" containerID="50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58"
Nov 01 12:23:42.660397 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.660343    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58"} err="failed to get container status \"50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58\": rpc error: code = NotFound desc = could not find container \"50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58\": container with ID starting with 50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58 not found: ID does not exist"
Nov 01 12:23:42.660397 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.660380    4095 scope.go:122] "RemoveContainer" containerID="cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf"
Nov 01 12:23:42.660647 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:42.660631    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf\": container with ID starting with cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf not found: ID does not exist" containerID="cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf"
Nov 01 12:23:42.660687 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.660655    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf"} err="failed to get container status \"cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf\": rpc error: code = NotFound desc = could not find container \"cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf\": container with ID starting with cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf not found: ID does not exist"
Nov 01 12:23:42.660687 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.660671    4095 scope.go:122] "RemoveContainer" containerID="71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0"
Nov 01 12:23:42.660878 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.660863    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0"} err="failed to get container status \"71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0\": rpc error: code = NotFound desc = could not find container \"71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0\": container with ID starting with 71920920ef0c301496face1fb51ab05055bb3e06bf0f82cfac5c7747463526b0 not found: ID does not exist"
Nov 01 12:23:42.660917 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.660878    4095 scope.go:122] "RemoveContainer" containerID="a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798"
Nov 01 12:23:42.661115 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.661097    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798"} err="failed to get container status \"a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798\": rpc error: code = NotFound desc = could not find container \"a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798\": container with ID starting with a08a807fa5bea59c8a94686f9e6966afffd5a906a0d6038243d0b3c58a28c798 not found: ID does not exist"
Nov 01 12:23:42.661151 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.661116    4095 scope.go:122] "RemoveContainer" containerID="d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2"
Nov 01 12:23:42.661334 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.661315    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2"} err="failed to get container status \"d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2\": rpc error: code = NotFound desc = could not find container \"d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2\": container with ID starting with d547ba1c48b42007e83e57c51ccfa1da94baad6ded84fd537f1407b68ef90da2 not found: ID does not exist"
Nov 01 12:23:42.661386 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.661335    4095 scope.go:122] "RemoveContainer" containerID="286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff"
Nov 01 12:23:42.661554 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.661535    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff"} err="failed to get container status \"286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff\": rpc error: code = NotFound desc = could not find container \"286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff\": container with ID starting with 286a978f316c218db3a9e6647c7d54b0392df9e16d82134a68b860ff9bf58eff not found: ID does not exist"
Nov 01 12:23:42.661597 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.661555    4095 scope.go:122] "RemoveContainer" containerID="81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c"
Nov 01 12:23:42.661769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.661739    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c"} err="failed to get container status \"81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c\": rpc error: code = NotFound desc = could not find container \"81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c\": container with ID starting with 81d26bc0aaf69c79d12c89054c06554d809cb8173cd95259d87a0eebf66f0c9c not found: ID does not exist"
Nov 01 12:23:42.661769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.661763    4095 scope.go:122] "RemoveContainer" containerID="50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58"
Nov 01 12:23:42.662049 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.662026    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58"} err="failed to get container status \"50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58\": rpc error: code = NotFound desc = could not find container \"50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58\": container with ID starting with 50ff01e0ae2d6c33a3501fdcb41c7747058cf9f59be46a4a2168ae0ed5a93b58 not found: ID does not exist"
Nov 01 12:23:42.662049 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.662047    4095 scope.go:122] "RemoveContainer" containerID="cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf"
Nov 01 12:23:42.662303 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.662285    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf"} err="failed to get container status \"cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf\": rpc error: code = NotFound desc = could not find container \"cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf\": container with ID starting with cda275ebd723251a3de9f4552b844577d9762ed42fc7a01cb04571af72755bdf not found: ID does not exist"
Nov 01 12:23:42.747920 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.747894    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 12:23:42.751338 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.751314    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 12:23:42.793644 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.793515    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6489db9f76-qs6gl" podUID="8941ba26-4008-4b09-be55-56c225259ec5" containerName="console" containerID="cri-o://c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918" gracePeriod=15
Nov 01 12:23:42.794662 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.794635    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 12:23:42.794990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.794956    4095 cpu_manager.go:414] "containerMap: removing container" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy-web"
Nov 01 12:23:42.794990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.794984    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy-web"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.794999    4095 cpu_manager.go:414] "containerMap: removing container" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="alertmanager"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795004    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="alertmanager"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795012    4095 cpu_manager.go:414] "containerMap: removing container" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="init-config-reloader"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795016    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="init-config-reloader"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795023    4095 cpu_manager.go:414] "containerMap: removing container" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy-metric"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795027    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy-metric"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795044    4095 cpu_manager.go:414] "containerMap: removing container" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="prom-label-proxy"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795049    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="prom-label-proxy"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795068    4095 cpu_manager.go:414] "containerMap: removing container" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy"
Nov 01 12:23:42.795073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795073    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy"
Nov 01 12:23:42.795337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795081    4095 cpu_manager.go:414] "containerMap: removing container" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="config-reloader"
Nov 01 12:23:42.795337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795085    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="config-reloader"
Nov 01 12:23:42.795337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795132    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy-web"
Nov 01 12:23:42.795337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795143    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="alertmanager"
Nov 01 12:23:42.795337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795150    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy-metric"
Nov 01 12:23:42.795337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795157    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="prom-label-proxy"
Nov 01 12:23:42.795337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795164    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="kube-rbac-proxy"
Nov 01 12:23:42.795337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.795169    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" containerName="config-reloader"
Nov 01 12:23:42.830340 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.830321    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.858263 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.858239    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 12:23:42.880889 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.880852    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Nov 01 12:23:42.880889 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.880873    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Nov 01 12:23:42.881099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.880912    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Nov 01 12:23:42.881099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.880959    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Nov 01 12:23:42.881099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.881027    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-tcjvl\""
Nov 01 12:23:42.881099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.881043    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Nov 01 12:23:42.881099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.881079    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Nov 01 12:23:42.881360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.881262    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Nov 01 12:23:42.881421 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.881410    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Nov 01 12:23:42.903987 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.903958    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Nov 01 12:23:42.918253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918232    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bcn\" (UniqueName: \"kubernetes.io/projected/fa9cd7ab-b91c-433f-bca3-6889327fa42c-kube-api-access-b6bcn\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918352 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918263    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa9cd7ab-b91c-433f-bca3-6889327fa42c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918352 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918288    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa9cd7ab-b91c-433f-bca3-6889327fa42c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918421 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918383    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa9cd7ab-b91c-433f-bca3-6889327fa42c-config-out\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918421 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918408    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918495 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918427    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918495 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918443    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918552    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918576    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918651 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918598    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918688 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918673    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9cd7ab-b91c-433f-bca3-6889327fa42c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918773 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918758    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-web-config\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:42.918823 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:42.918791    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa9cd7ab-b91c-433f-bca3-6889327fa42c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019453 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019422    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa9cd7ab-b91c-433f-bca3-6889327fa42c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019621 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019482    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bcn\" (UniqueName: \"kubernetes.io/projected/fa9cd7ab-b91c-433f-bca3-6889327fa42c-kube-api-access-b6bcn\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019621 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019500    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa9cd7ab-b91c-433f-bca3-6889327fa42c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019621 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019515    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa9cd7ab-b91c-433f-bca3-6889327fa42c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019621 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019551    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa9cd7ab-b91c-433f-bca3-6889327fa42c-config-out\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019621 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019574    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019621 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019605    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019626    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019657    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019682    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019708    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019735    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9cd7ab-b91c-433f-bca3-6889327fa42c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.019944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.019763    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-web-config\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.020290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.020000    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fa9cd7ab-b91c-433f-bca3-6889327fa42c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.020365 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.020337    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fa9cd7ab-b91c-433f-bca3-6889327fa42c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.021655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.021631    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fa9cd7ab-b91c-433f-bca3-6889327fa42c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.022841 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.022810    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fa9cd7ab-b91c-433f-bca3-6889327fa42c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.022953 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.022820    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-config-volume\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.023493 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.023158    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.023493 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.023187    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-web-config\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.023493 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.023274    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fa9cd7ab-b91c-433f-bca3-6889327fa42c-config-out\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.023493 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.023442    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.024318 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.024292    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.024558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.024542    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.024808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.024793    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fa9cd7ab-b91c-433f-bca3-6889327fa42c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.073608 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.073581    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bcn\" (UniqueName: \"kubernetes.io/projected/fa9cd7ab-b91c-433f-bca3-6889327fa42c-kube-api-access-b6bcn\") pod \"alertmanager-main-0\" (UID: \"fa9cd7ab-b91c-433f-bca3-6889327fa42c\") " pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.083938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.083918    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:23:43.120690 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.120659    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-oauth-config\") pod \"8941ba26-4008-4b09-be55-56c225259ec5\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") "
Nov 01 12:23:43.120690 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.120694    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8941ba26-4008-4b09-be55-56c225259ec5-kube-api-access-nkgfx\" (UniqueName: \"kubernetes.io/projected/8941ba26-4008-4b09-be55-56c225259ec5-kube-api-access-nkgfx\") pod \"8941ba26-4008-4b09-be55-56c225259ec5\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") "
Nov 01 12:23:43.120848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.120728    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-serving-cert\") pod \"8941ba26-4008-4b09-be55-56c225259ec5\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") "
Nov 01 12:23:43.120848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.120769    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-console-config\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-console-config\") pod \"8941ba26-4008-4b09-be55-56c225259ec5\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") "
Nov 01 12:23:43.120848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.120810    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8941ba26-4008-4b09-be55-56c225259ec5-tmp\" (UniqueName: \"kubernetes.io/empty-dir/8941ba26-4008-4b09-be55-56c225259ec5-tmp\") pod \"8941ba26-4008-4b09-be55-56c225259ec5\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") "
Nov 01 12:23:43.121021 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.120852    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-service-ca\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-service-ca\") pod \"8941ba26-4008-4b09-be55-56c225259ec5\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") "
Nov 01 12:23:43.121021 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.120917    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-oauth-serving-cert\") pod \"8941ba26-4008-4b09-be55-56c225259ec5\" (UID: \"8941ba26-4008-4b09-be55-56c225259ec5\") "
Nov 01 12:23:43.121786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.121338    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-service-ca" pod "8941ba26-4008-4b09-be55-56c225259ec5" (UID: "8941ba26-4008-4b09-be55-56c225259ec5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:43.121786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.121374    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-oauth-serving-cert" pod "8941ba26-4008-4b09-be55-56c225259ec5" (UID: "8941ba26-4008-4b09-be55-56c225259ec5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:43.121786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.121380    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8941ba26-4008-4b09-be55-56c225259ec5-tmp" pod "8941ba26-4008-4b09-be55-56c225259ec5" (UID: "8941ba26-4008-4b09-be55-56c225259ec5"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:23:43.121786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.121749    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-console-config" pod "8941ba26-4008-4b09-be55-56c225259ec5" (UID: "8941ba26-4008-4b09-be55-56c225259ec5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:43.122099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.121921    4095 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8941ba26-4008-4b09-be55-56c225259ec5-tmp\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:43.122099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.121942    4095 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:43.122099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.121958    4095 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-oauth-serving-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:43.123273 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.123252    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-serving-cert" pod "8941ba26-4008-4b09-be55-56c225259ec5" (UID: "8941ba26-4008-4b09-be55-56c225259ec5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:43.123405 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.123385    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-oauth-config" pod "8941ba26-4008-4b09-be55-56c225259ec5" (UID: "8941ba26-4008-4b09-be55-56c225259ec5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:43.123405 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.123394    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8941ba26-4008-4b09-be55-56c225259ec5-kube-api-access-nkgfx" pod "8941ba26-4008-4b09-be55-56c225259ec5" (UID: "8941ba26-4008-4b09-be55-56c225259ec5"). InnerVolumeSpecName "kube-api-access-nkgfx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:23:43.223211 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.223169    4095 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-oauth-config\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:43.223211 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.223200    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkgfx\" (UniqueName: \"kubernetes.io/projected/8941ba26-4008-4b09-be55-56c225259ec5-kube-api-access-nkgfx\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:43.223211 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.223215    4095 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8941ba26-4008-4b09-be55-56c225259ec5-console-serving-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:43.223435 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.223230    4095 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8941ba26-4008-4b09-be55-56c225259ec5-console-config\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:43.346112 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.346023    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Nov 01 12:23:43.609148 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.609119    4095 generic.go:356] "Generic (PLEG): container finished" podID="8941ba26-4008-4b09-be55-56c225259ec5" containerID="c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918" exitCode=0
Nov 01 12:23:43.609599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.609167    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6489db9f76-qs6gl"
Nov 01 12:23:43.609599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.609205    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6489db9f76-qs6gl" event={"ID":"8941ba26-4008-4b09-be55-56c225259ec5","Type":"ContainerDied","Data":"c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918"}
Nov 01 12:23:43.609599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.609254    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6489db9f76-qs6gl" event={"ID":"8941ba26-4008-4b09-be55-56c225259ec5","Type":"ContainerDied","Data":"760e66a8b88d25743573ef8c65c5fea5d37ad45e76a5671e6b292dc9f31c9b10"}
Nov 01 12:23:43.609599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.609271    4095 scope.go:122] "RemoveContainer" containerID="c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918"
Nov 01 12:23:43.617643 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.617616    4095 scope.go:122] "RemoveContainer" containerID="c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918"
Nov 01 12:23:43.617921 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:43.617887    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918\": container with ID starting with c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918 not found: ID does not exist" containerID="c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918"
Nov 01 12:23:43.618008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.617928    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918"} err="failed to get container status \"c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918\": rpc error: code = NotFound desc = could not find container \"c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918\": container with ID starting with c76c5143d6981ffcabf0b68e256c1776dcc01425be17efedd9b6f354ec561918 not found: ID does not exist"
Nov 01 12:23:43.765058 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.765029    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6489db9f76-qs6gl"]
Nov 01 12:23:43.768452 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.768431    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6489db9f76-qs6gl"]
Nov 01 12:23:43.837519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:43.837496    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Nov 01 12:23:43.839430 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:23:43.839396    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9cd7ab_b91c_433f_bca3_6889327fa42c.slice/crio-d5703572a04d9e73c818ad9d8f95b94b2184c5fedc2754d5239ce1ff30e9e435 WatchSource:0}: Error finding container d5703572a04d9e73c818ad9d8f95b94b2184c5fedc2754d5239ce1ff30e9e435: Status 404 returned error can't find the container with id d5703572a04d9e73c818ad9d8f95b94b2184c5fedc2754d5239ce1ff30e9e435
Nov 01 12:23:44.186544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.186471    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8941ba26-4008-4b09-be55-56c225259ec5" path="/var/lib/kubelet/pods/8941ba26-4008-4b09-be55-56c225259ec5/volumes"
Nov 01 12:23:44.186912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.186898    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da18aad7-927f-4f7f-baca-07ff38303dd1" path="/var/lib/kubelet/pods/da18aad7-927f-4f7f-baca-07ff38303dd1/volumes"
Nov 01 12:23:44.616837 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.616800    4095 generic.go:356] "Generic (PLEG): container finished" podID="fa9cd7ab-b91c-433f-bca3-6889327fa42c" containerID="f13788b87de8181c9a652f754bbc8ded1d9eda2f74595bddcf8ffafad035c69e" exitCode=0
Nov 01 12:23:44.617346 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.616928    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9cd7ab-b91c-433f-bca3-6889327fa42c","Type":"ContainerDied","Data":"f13788b87de8181c9a652f754bbc8ded1d9eda2f74595bddcf8ffafad035c69e"}
Nov 01 12:23:44.617346 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.617020    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9cd7ab-b91c-433f-bca3-6889327fa42c","Type":"ContainerStarted","Data":"d5703572a04d9e73c818ad9d8f95b94b2184c5fedc2754d5239ce1ff30e9e435"}
Nov 01 12:23:44.833105 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.833078    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"]
Nov 01 12:23:44.833708 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.833691    4095 cpu_manager.go:414] "containerMap: removing container" podUID="8941ba26-4008-4b09-be55-56c225259ec5" containerName="console"
Nov 01 12:23:44.833815 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.833709    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8941ba26-4008-4b09-be55-56c225259ec5" containerName="console"
Nov 01 12:23:44.833903 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.833889    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="8941ba26-4008-4b09-be55-56c225259ec5" containerName="console"
Nov 01 12:23:44.853715 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.853689    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:44.903818 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.903781    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 01 12:23:44.903958 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.903881    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 01 12:23:44.903958 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.903902    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 01 12:23:44.903958 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.903924    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-2hx2b\""
Nov 01 12:23:44.904155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.904000    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 01 12:23:44.904214 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.904200    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 01 12:23:44.936338 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.936310    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:44.936497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.936361    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4289e74e-8746-4732-9ffd-70b4e7c746c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:44.936497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.936471    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-federate-client-tls\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:44.936739 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.936499    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-telemeter-client-tls\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:44.936739 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.936576    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-secret-telemeter-client\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:44.936739 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.936596    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4289e74e-8746-4732-9ffd-70b4e7c746c4-metrics-client-ca\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:44.936739 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.936634    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4289e74e-8746-4732-9ffd-70b4e7c746c4-serving-certs-ca-bundle\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:44.936739 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.936659    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hptgh\" (UniqueName: \"kubernetes.io/projected/4289e74e-8746-4732-9ffd-70b4e7c746c4-kube-api-access-hptgh\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:44.951851 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.951824    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 01 12:23:44.953017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:44.953000    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"]
Nov 01 12:23:45.037774 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.037752    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4289e74e-8746-4732-9ffd-70b4e7c746c4-serving-certs-ca-bundle\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.037877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.037783    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hptgh\" (UniqueName: \"kubernetes.io/projected/4289e74e-8746-4732-9ffd-70b4e7c746c4-kube-api-access-hptgh\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.037877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.037808    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.038035 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.038014    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4289e74e-8746-4732-9ffd-70b4e7c746c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.038108 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.038089    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-federate-client-tls\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.038185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.038133    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-telemeter-client-tls\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.038238 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.038189    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-secret-telemeter-client\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.038238 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.038214    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4289e74e-8746-4732-9ffd-70b4e7c746c4-metrics-client-ca\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.038609 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.038582    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4289e74e-8746-4732-9ffd-70b4e7c746c4-serving-certs-ca-bundle\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.038930 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.038908    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4289e74e-8746-4732-9ffd-70b4e7c746c4-metrics-client-ca\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.039037 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.038951    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4289e74e-8746-4732-9ffd-70b4e7c746c4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.040254 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.040224    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.040409 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.040394    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-telemeter-client-tls\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.040465 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.040447    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-federate-client-tls\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.040523 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.040510    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/4289e74e-8746-4732-9ffd-70b4e7c746c4-secret-telemeter-client\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.091850 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.091827    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hptgh\" (UniqueName: \"kubernetes.io/projected/4289e74e-8746-4732-9ffd-70b4e7c746c4-kube-api-access-hptgh\") pod \"telemeter-client-6c749d9f6b-86ntp\" (UID: \"4289e74e-8746-4732-9ffd-70b4e7c746c4\") " pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.226929 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.226909    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"
Nov 01 12:23:45.626358 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.626323    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9cd7ab-b91c-433f-bca3-6889327fa42c","Type":"ContainerStarted","Data":"b588de0ae9ca969dba0d07506b0e110507f0c89d84fe0f353c4dbc0ea0a4a534"}
Nov 01 12:23:45.626771 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.626405    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9cd7ab-b91c-433f-bca3-6889327fa42c","Type":"ContainerStarted","Data":"ba3bc3a625729f3f55b7ce08f7fb8e6286f7dd50b6a57d82d70e968a82f3aa6a"}
Nov 01 12:23:45.626771 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.626444    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9cd7ab-b91c-433f-bca3-6889327fa42c","Type":"ContainerStarted","Data":"fd19101fe94d97810ba7eb599adc44553bff0e2af197495dea5210bd1890e300"}
Nov 01 12:23:45.626771 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.626480    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9cd7ab-b91c-433f-bca3-6889327fa42c","Type":"ContainerStarted","Data":"42914113d8a4434ace388eca7e075d8a33a9244dcf2c4411aded4c29f733498c"}
Nov 01 12:23:45.626771 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.626515    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9cd7ab-b91c-433f-bca3-6889327fa42c","Type":"ContainerStarted","Data":"e43fd178d4adb9452b32aacb3aa8c660aaee9cf50f7d2567958bf18c9a421c9f"}
Nov 01 12:23:45.626771 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.626550    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fa9cd7ab-b91c-433f-bca3-6889327fa42c","Type":"ContainerStarted","Data":"25e6dea05829c6c3ff1e24221b5e7e4586e961c96937d079ddc1841ad5f9cb88"}
Nov 01 12:23:45.706724 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.706690    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c749d9f6b-86ntp"]
Nov 01 12:23:45.708583 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:23:45.708555    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4289e74e_8746_4732_9ffd_70b4e7c746c4.slice/crio-63dd05d39fff2e4601ae2597e9997dc660e0d1afa0d94bf56c523802b8cff529 WatchSource:0}: Error finding container 63dd05d39fff2e4601ae2597e9997dc660e0d1afa0d94bf56c523802b8cff529: Status 404 returned error can't find the container with id 63dd05d39fff2e4601ae2597e9997dc660e0d1afa0d94bf56c523802b8cff529
Nov 01 12:23:45.744124 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:45.744067    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.731398768 podStartE2EDuration="3.744048708s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.64994ms" isStatefulPod=false podCreationTimestamp="2026-11-01 12:23:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:23:45.743296438 +0000 UTC m=+85.661092143" watchObservedRunningTime="2026-11-01 12:23:45.744048708 +0000 UTC m=+85.661844375"
Nov 01 12:23:46.631196 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:46.631164    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp" event={"ID":"4289e74e-8746-4732-9ffd-70b4e7c746c4","Type":"ContainerStarted","Data":"63dd05d39fff2e4601ae2597e9997dc660e0d1afa0d94bf56c523802b8cff529"}
Nov 01 12:23:46.732274 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:46.732243    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 12:23:46.732542 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:46.732513    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="prometheus" containerID="cri-o://4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d" gracePeriod=600
Nov 01 12:23:46.732781 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:46.732553    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044" gracePeriod=600
Nov 01 12:23:46.732883 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:46.732568    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="thanos-sidecar" containerID="cri-o://3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964" gracePeriod=600
Nov 01 12:23:46.732940 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:46.732542    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy" containerID="cri-o://62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112" gracePeriod=600
Nov 01 12:23:46.733005 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:46.732576    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy-web" containerID="cri-o://00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972" gracePeriod=600
Nov 01 12:23:46.733071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:46.732584    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="config-reloader" containerID="cri-o://613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433" gracePeriod=600
Nov 01 12:23:47.040572 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.040543    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.158308 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158271    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158323    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-tls-assets\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-tls-assets\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158361    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158403    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-1\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158429    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-grpc-tls\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158462    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-web-config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-web-config\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158486    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-trusted-ca-bundle\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158800 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158518    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-kubelet-serving-ca-bundle\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158800 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158566    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-metrics-client-ca\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158800 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158599    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-kube-rbac-proxy\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158800 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158636    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-metrics-client-certs\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158800 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158668    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158800 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158695    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config-out\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158800 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158742    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-2\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.158800 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158766    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-tls\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.159266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158803    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-serving-certs-ca-bundle\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.159266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158857    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-db\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.159266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158889    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-kube-api-access-nwtvr\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-kube-api-access-nwtvr\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.159266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.158996    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-thanos-prometheus-http-client-file\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.159266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.159030    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-0\") pod \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\" (UID: \"2aa78254-1892-4d5f-83f3-77abc7a5e39c\") "
Nov 01 12:23:47.161013 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.160196    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-kubelet-serving-ca-bundle" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:47.161013 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.160648    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-trusted-ca-bundle" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:47.161013 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.160710    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-0" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:47.161590 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.161557    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-serving-certs-ca-bundle" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:47.162430 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.162400    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-db" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:23:47.164401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.163511    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-tls-assets" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:23:47.164401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.163541    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-kube-rbac-proxy-web" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:47.164401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.163654    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:47.164401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.163728    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-kube-rbac-proxy" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:47.164401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.164181    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-tls" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:47.164735 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.164413    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-metrics-client-ca" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:47.164799 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.164739    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-2" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:47.165067 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.165043    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-1" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:23:47.166515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.166488    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-thanos-sidecar-tls" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:47.167319 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.167292    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-kube-api-access-nwtvr" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "kube-api-access-nwtvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:23:47.168151 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.168070    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-thanos-prometheus-http-client-file" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:47.168151 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.168133    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-metrics-client-certs" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:47.168760 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.168735    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-grpc-tls" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:47.168841 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.168814    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config-out" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:23:47.176758 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.176729    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-web-config" pod "2aa78254-1892-4d5f-83f3-77abc7a5e39c" (UID: "2aa78254-1892-4d5f-83f3-77abc7a5e39c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:23:47.259816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259753    4095 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-thanos-prometheus-http-client-file\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.259816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259777    4095 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.259816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259787    4095 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.259816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259798    4095 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-tls-assets\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.259816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259807    4095 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.259816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259816    4095 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259828    4095 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-grpc-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259843    4095 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-web-config\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259856    4095 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259869    4095 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259879    4095 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-metrics-client-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259887    4095 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-kube-rbac-proxy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259896    4095 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-metrics-client-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259906    4095 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259914    4095 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-config-out\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259922    4095 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259935    4095 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/2aa78254-1892-4d5f-83f3-77abc7a5e39c-secret-prometheus-k8s-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259949    4095 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2aa78254-1892-4d5f-83f3-77abc7a5e39c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259964    4095 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/2aa78254-1892-4d5f-83f3-77abc7a5e39c-prometheus-k8s-db\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.260289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.259997    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwtvr\" (UniqueName: \"kubernetes.io/projected/2aa78254-1892-4d5f-83f3-77abc7a5e39c-kube-api-access-nwtvr\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:23:47.640146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640110    4095 generic.go:356] "Generic (PLEG): container finished" podID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerID="1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044" exitCode=0
Nov 01 12:23:47.640146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640144    4095 generic.go:356] "Generic (PLEG): container finished" podID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerID="62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112" exitCode=0
Nov 01 12:23:47.640146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640155    4095 generic.go:356] "Generic (PLEG): container finished" podID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerID="00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972" exitCode=0
Nov 01 12:23:47.640146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640164    4095 generic.go:356] "Generic (PLEG): container finished" podID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerID="3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964" exitCode=0
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640173    4095 generic.go:356] "Generic (PLEG): container finished" podID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerID="613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433" exitCode=0
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640178    4095 generic.go:356] "Generic (PLEG): container finished" podID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerID="4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d" exitCode=0
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640177    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerDied","Data":"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"}
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640222    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640273    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerDied","Data":"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"}
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640315    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerDied","Data":"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"}
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640340    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerDied","Data":"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"}
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640381    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerDied","Data":"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"}
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640421    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerDied","Data":"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"}
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640457    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"2aa78254-1892-4d5f-83f3-77abc7a5e39c","Type":"ContainerDied","Data":"8b6a2a4793bee57bc8faf5bd135ed515869ddc71035b4db716710e15250676f1"}
Nov 01 12:23:47.640766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.640382    4095 scope.go:122] "RemoveContainer" containerID="1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"
Nov 01 12:23:47.650216 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.650200    4095 scope.go:122] "RemoveContainer" containerID="62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"
Nov 01 12:23:47.658536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.658517    4095 scope.go:122] "RemoveContainer" containerID="00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"
Nov 01 12:23:47.666391 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.666370    4095 scope.go:122] "RemoveContainer" containerID="3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"
Nov 01 12:23:47.675449 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.675373    4095 scope.go:122] "RemoveContainer" containerID="613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"
Nov 01 12:23:47.686094 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.686073    4095 scope.go:122] "RemoveContainer" containerID="4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"
Nov 01 12:23:47.696743 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.696517    4095 scope.go:122] "RemoveContainer" containerID="a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"
Nov 01 12:23:47.706958 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.706936    4095 scope.go:122] "RemoveContainer" containerID="1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"
Nov 01 12:23:47.707303 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:47.707282    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": container with ID starting with 1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044 not found: ID does not exist" containerID="1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"
Nov 01 12:23:47.707383 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.707313    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"} err="failed to get container status \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": rpc error: code = NotFound desc = could not find container \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": container with ID starting with 1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044 not found: ID does not exist"
Nov 01 12:23:47.707383 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.707339    4095 scope.go:122] "RemoveContainer" containerID="62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"
Nov 01 12:23:47.707642 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:47.707618    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": container with ID starting with 62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112 not found: ID does not exist" containerID="62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"
Nov 01 12:23:47.707746 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.707652    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"} err="failed to get container status \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": rpc error: code = NotFound desc = could not find container \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": container with ID starting with 62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112 not found: ID does not exist"
Nov 01 12:23:47.707746 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.707675    4095 scope.go:122] "RemoveContainer" containerID="00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"
Nov 01 12:23:47.708316 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:47.707938    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": container with ID starting with 00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972 not found: ID does not exist" containerID="00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"
Nov 01 12:23:47.708316 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.707993    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"} err="failed to get container status \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": rpc error: code = NotFound desc = could not find container \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": container with ID starting with 00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972 not found: ID does not exist"
Nov 01 12:23:47.708316 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.708016    4095 scope.go:122] "RemoveContainer" containerID="3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"
Nov 01 12:23:47.708316 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:47.708287    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": container with ID starting with 3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964 not found: ID does not exist" containerID="3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"
Nov 01 12:23:47.708559 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.708312    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"} err="failed to get container status \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": rpc error: code = NotFound desc = could not find container \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": container with ID starting with 3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964 not found: ID does not exist"
Nov 01 12:23:47.708559 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.708334    4095 scope.go:122] "RemoveContainer" containerID="613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"
Nov 01 12:23:47.708668 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:47.708602    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": container with ID starting with 613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433 not found: ID does not exist" containerID="613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"
Nov 01 12:23:47.708668 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.708630    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"} err="failed to get container status \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": rpc error: code = NotFound desc = could not find container \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": container with ID starting with 613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433 not found: ID does not exist"
Nov 01 12:23:47.708668 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.708649    4095 scope.go:122] "RemoveContainer" containerID="4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"
Nov 01 12:23:47.708906 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:47.708886    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": container with ID starting with 4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d not found: ID does not exist" containerID="4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"
Nov 01 12:23:47.709000 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.708908    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"} err="failed to get container status \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": rpc error: code = NotFound desc = could not find container \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": container with ID starting with 4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d not found: ID does not exist"
Nov 01 12:23:47.709000 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.708923    4095 scope.go:122] "RemoveContainer" containerID="a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"
Nov 01 12:23:47.709234 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:47.709216    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": container with ID starting with a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2 not found: ID does not exist" containerID="a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"
Nov 01 12:23:47.709314 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.709239    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"} err="failed to get container status \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": rpc error: code = NotFound desc = could not find container \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": container with ID starting with a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2 not found: ID does not exist"
Nov 01 12:23:47.709314 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.709256    4095 scope.go:122] "RemoveContainer" containerID="1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"
Nov 01 12:23:47.709604 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.709580    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"} err="failed to get container status \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": rpc error: code = NotFound desc = could not find container \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": container with ID starting with 1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044 not found: ID does not exist"
Nov 01 12:23:47.709679 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.709603    4095 scope.go:122] "RemoveContainer" containerID="62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"
Nov 01 12:23:47.709869 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.709845    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"} err="failed to get container status \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": rpc error: code = NotFound desc = could not find container \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": container with ID starting with 62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112 not found: ID does not exist"
Nov 01 12:23:47.709928 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.709873    4095 scope.go:122] "RemoveContainer" containerID="00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"
Nov 01 12:23:47.710191 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.710166    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"} err="failed to get container status \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": rpc error: code = NotFound desc = could not find container \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": container with ID starting with 00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972 not found: ID does not exist"
Nov 01 12:23:47.710289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.710191    4095 scope.go:122] "RemoveContainer" containerID="3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"
Nov 01 12:23:47.710596 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.710494    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"} err="failed to get container status \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": rpc error: code = NotFound desc = could not find container \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": container with ID starting with 3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964 not found: ID does not exist"
Nov 01 12:23:47.710596 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.710528    4095 scope.go:122] "RemoveContainer" containerID="613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"
Nov 01 12:23:47.710911 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.710873    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"} err="failed to get container status \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": rpc error: code = NotFound desc = could not find container \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": container with ID starting with 613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433 not found: ID does not exist"
Nov 01 12:23:47.710911 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.710901    4095 scope.go:122] "RemoveContainer" containerID="4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"
Nov 01 12:23:47.711319 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.711294    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"} err="failed to get container status \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": rpc error: code = NotFound desc = could not find container \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": container with ID starting with 4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d not found: ID does not exist"
Nov 01 12:23:47.711382 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.711320    4095 scope.go:122] "RemoveContainer" containerID="a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"
Nov 01 12:23:47.711639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.711617    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"} err="failed to get container status \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": rpc error: code = NotFound desc = could not find container \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": container with ID starting with a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2 not found: ID does not exist"
Nov 01 12:23:47.711702 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.711641    4095 scope.go:122] "RemoveContainer" containerID="1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"
Nov 01 12:23:47.712860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.711996    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"} err="failed to get container status \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": rpc error: code = NotFound desc = could not find container \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": container with ID starting with 1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044 not found: ID does not exist"
Nov 01 12:23:47.712860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.712020    4095 scope.go:122] "RemoveContainer" containerID="62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"
Nov 01 12:23:47.712860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.712321    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"} err="failed to get container status \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": rpc error: code = NotFound desc = could not find container \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": container with ID starting with 62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112 not found: ID does not exist"
Nov 01 12:23:47.712860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.712339    4095 scope.go:122] "RemoveContainer" containerID="00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"
Nov 01 12:23:47.712860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.712650    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"} err="failed to get container status \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": rpc error: code = NotFound desc = could not find container \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": container with ID starting with 00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972 not found: ID does not exist"
Nov 01 12:23:47.712860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.712673    4095 scope.go:122] "RemoveContainer" containerID="3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"
Nov 01 12:23:47.713187 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.712951    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"} err="failed to get container status \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": rpc error: code = NotFound desc = could not find container \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": container with ID starting with 3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964 not found: ID does not exist"
Nov 01 12:23:47.713187 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.712992    4095 scope.go:122] "RemoveContainer" containerID="613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"
Nov 01 12:23:47.713429 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.713407    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"} err="failed to get container status \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": rpc error: code = NotFound desc = could not find container \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": container with ID starting with 613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433 not found: ID does not exist"
Nov 01 12:23:47.713495 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.713439    4095 scope.go:122] "RemoveContainer" containerID="4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"
Nov 01 12:23:47.713702 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.713681    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"} err="failed to get container status \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": rpc error: code = NotFound desc = could not find container \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": container with ID starting with 4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d not found: ID does not exist"
Nov 01 12:23:47.713702 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.713700    4095 scope.go:122] "RemoveContainer" containerID="a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"
Nov 01 12:23:47.714004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.713954    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"} err="failed to get container status \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": rpc error: code = NotFound desc = could not find container \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": container with ID starting with a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2 not found: ID does not exist"
Nov 01 12:23:47.714120 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.714006    4095 scope.go:122] "RemoveContainer" containerID="1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"
Nov 01 12:23:47.714294 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.714272    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"} err="failed to get container status \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": rpc error: code = NotFound desc = could not find container \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": container with ID starting with 1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044 not found: ID does not exist"
Nov 01 12:23:47.714372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.714294    4095 scope.go:122] "RemoveContainer" containerID="62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"
Nov 01 12:23:47.714633 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.714576    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"} err="failed to get container status \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": rpc error: code = NotFound desc = could not find container \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": container with ID starting with 62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112 not found: ID does not exist"
Nov 01 12:23:47.714711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.714635    4095 scope.go:122] "RemoveContainer" containerID="00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"
Nov 01 12:23:47.714907 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.714886    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"} err="failed to get container status \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": rpc error: code = NotFound desc = could not find container \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": container with ID starting with 00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972 not found: ID does not exist"
Nov 01 12:23:47.714907 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.714906    4095 scope.go:122] "RemoveContainer" containerID="3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"
Nov 01 12:23:47.715221 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.715193    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"} err="failed to get container status \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": rpc error: code = NotFound desc = could not find container \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": container with ID starting with 3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964 not found: ID does not exist"
Nov 01 12:23:47.715295 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.715228    4095 scope.go:122] "RemoveContainer" containerID="613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"
Nov 01 12:23:47.715503 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.715484    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"} err="failed to get container status \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": rpc error: code = NotFound desc = could not find container \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": container with ID starting with 613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433 not found: ID does not exist"
Nov 01 12:23:47.715503 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.715502    4095 scope.go:122] "RemoveContainer" containerID="4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"
Nov 01 12:23:47.715758 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.715741    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"} err="failed to get container status \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": rpc error: code = NotFound desc = could not find container \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": container with ID starting with 4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d not found: ID does not exist"
Nov 01 12:23:47.715758 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.715757    4095 scope.go:122] "RemoveContainer" containerID="a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"
Nov 01 12:23:47.716050 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.716023    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"} err="failed to get container status \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": rpc error: code = NotFound desc = could not find container \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": container with ID starting with a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2 not found: ID does not exist"
Nov 01 12:23:47.716050 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.716049    4095 scope.go:122] "RemoveContainer" containerID="1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"
Nov 01 12:23:47.716305 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.716278    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"} err="failed to get container status \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": rpc error: code = NotFound desc = could not find container \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": container with ID starting with 1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044 not found: ID does not exist"
Nov 01 12:23:47.716305 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.716309    4095 scope.go:122] "RemoveContainer" containerID="62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"
Nov 01 12:23:47.716587 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.716562    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"} err="failed to get container status \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": rpc error: code = NotFound desc = could not find container \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": container with ID starting with 62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112 not found: ID does not exist"
Nov 01 12:23:47.716587 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.716587    4095 scope.go:122] "RemoveContainer" containerID="00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"
Nov 01 12:23:47.716875 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.716851    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"} err="failed to get container status \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": rpc error: code = NotFound desc = could not find container \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": container with ID starting with 00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972 not found: ID does not exist"
Nov 01 12:23:47.716943 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.716875    4095 scope.go:122] "RemoveContainer" containerID="3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"
Nov 01 12:23:47.717204 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.717177    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"} err="failed to get container status \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": rpc error: code = NotFound desc = could not find container \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": container with ID starting with 3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964 not found: ID does not exist"
Nov 01 12:23:47.717204 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.717196    4095 scope.go:122] "RemoveContainer" containerID="613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"
Nov 01 12:23:47.717447 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.717424    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"} err="failed to get container status \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": rpc error: code = NotFound desc = could not find container \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": container with ID starting with 613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433 not found: ID does not exist"
Nov 01 12:23:47.717522 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.717448    4095 scope.go:122] "RemoveContainer" containerID="4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"
Nov 01 12:23:47.717766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.717745    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"} err="failed to get container status \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": rpc error: code = NotFound desc = could not find container \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": container with ID starting with 4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d not found: ID does not exist"
Nov 01 12:23:47.717850 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.717766    4095 scope.go:122] "RemoveContainer" containerID="a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"
Nov 01 12:23:47.718035 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.718011    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"} err="failed to get container status \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": rpc error: code = NotFound desc = could not find container \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": container with ID starting with a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2 not found: ID does not exist"
Nov 01 12:23:47.718035 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.718034    4095 scope.go:122] "RemoveContainer" containerID="1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"
Nov 01 12:23:47.718319 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.718295    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044"} err="failed to get container status \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": rpc error: code = NotFound desc = could not find container \"1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044\": container with ID starting with 1d0c8ff63fe54765ae1943a55cee55674566b082de865e011ff79d4f48ebd044 not found: ID does not exist"
Nov 01 12:23:47.718384 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.718321    4095 scope.go:122] "RemoveContainer" containerID="62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"
Nov 01 12:23:47.718561 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.718538    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112"} err="failed to get container status \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": rpc error: code = NotFound desc = could not find container \"62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112\": container with ID starting with 62b24908e01ca27a875e623ecd3b97d337fcccdc4f6443312481adb56359d112 not found: ID does not exist"
Nov 01 12:23:47.718622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.718562    4095 scope.go:122] "RemoveContainer" containerID="00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"
Nov 01 12:23:47.718804 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.718784    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972"} err="failed to get container status \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": rpc error: code = NotFound desc = could not find container \"00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972\": container with ID starting with 00376be5f969e4782c51d33d74e04f1bfb59f60b4a28a749cdbd966ab91b0972 not found: ID does not exist"
Nov 01 12:23:47.718804 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.718803    4095 scope.go:122] "RemoveContainer" containerID="3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"
Nov 01 12:23:47.719071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.719053    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964"} err="failed to get container status \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": rpc error: code = NotFound desc = could not find container \"3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964\": container with ID starting with 3cb282bc5f895cda6ecb0402bf2eec90af7c28b9c55fd67a4472e657ab139964 not found: ID does not exist"
Nov 01 12:23:47.719152 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.719074    4095 scope.go:122] "RemoveContainer" containerID="613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"
Nov 01 12:23:47.719338 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.719321    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433"} err="failed to get container status \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": rpc error: code = NotFound desc = could not find container \"613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433\": container with ID starting with 613b814eb57135cba9350dd4c8a6824a5099e7227289c4507cab464aeecfe433 not found: ID does not exist"
Nov 01 12:23:47.719415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.719338    4095 scope.go:122] "RemoveContainer" containerID="4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"
Nov 01 12:23:47.719584 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.719567    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d"} err="failed to get container status \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": rpc error: code = NotFound desc = could not find container \"4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d\": container with ID starting with 4255477e8353f0b6f0fd0bc3a1be192a448987996e6256d31df1648fee81f13d not found: ID does not exist"
Nov 01 12:23:47.719656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.719584    4095 scope.go:122] "RemoveContainer" containerID="a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"
Nov 01 12:23:47.719875 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.719850    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2"} err="failed to get container status \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": rpc error: code = NotFound desc = could not find container \"a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2\": container with ID starting with a9eed9bf49c2837377f174a56d024309a316373b09c12fc071b5ae9016c45fb2 not found: ID does not exist"
Nov 01 12:23:47.804479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.804445    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 12:23:47.851429 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.851397    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 12:23:47.852258 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852236    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852575    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852593    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852608    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="config-reloader"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852614    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="config-reloader"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852620    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="thanos-sidecar"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852626    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="thanos-sidecar"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852632    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy-thanos"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852637    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy-thanos"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852644    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="init-config-reloader"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852649    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="init-config-reloader"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852655    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="prometheus"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852663    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="prometheus"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852672    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy-web"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852677    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy-web"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852731    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852742    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="config-reloader"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852751    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="thanos-sidecar"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852759    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy-thanos"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852767    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="kube-rbac-proxy-web"
Nov 01 12:23:47.854009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.852776    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" containerName="prometheus"
Nov 01 12:23:47.884181 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.884156    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.936558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.936517    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 01 12:23:47.936924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.936866    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 01 12:23:47.936924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.936868    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 01 12:23:47.936924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.936923    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 01 12:23:47.937218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.936936    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 01 12:23:47.937218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.937003    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 01 12:23:47.937218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.937087    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 01 12:23:47.937218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.937133    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 01 12:23:47.937218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.937226    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 01 12:23:47.937548 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.937286    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-scdt8\""
Nov 01 12:23:47.937548 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.937380    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e8109ldfulpk9\""
Nov 01 12:23:47.958580 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.958541    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 12:23:47.965283 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965248    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965291    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cccc1506-3c47-4a60-995b-557c7ed41a65-config-out\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965318    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965361    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965435    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965556    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965581    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965599    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965622    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlt8l\" (UniqueName: \"kubernetes.io/projected/cccc1506-3c47-4a60-995b-557c7ed41a65-kube-api-access-mlt8l\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965821 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965686    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-config\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965821 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965715    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965821 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965735    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965821 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965749    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965930 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965799    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.965930 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965913    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cccc1506-3c47-4a60-995b-557c7ed41a65-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.966019 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965944    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.966019 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.965996    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.966120 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.966022    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-web-config\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.966120 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.966050    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.966120 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.966098    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:47.966257 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.966139    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 01 12:23:47.966257 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.966142    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 01 12:23:47.966257 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:47.966171    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 01 12:23:48.014892 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.014854    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 01 12:23:48.015149 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.015133    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 01 12:23:48.066872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.066835    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.066872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.066872    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cccc1506-3c47-4a60-995b-557c7ed41a65-config-out\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067110 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.066888    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067110 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.066914    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067110 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.066940    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067110 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.066987    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067110 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067013    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067149    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067188    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlt8l\" (UniqueName: \"kubernetes.io/projected/cccc1506-3c47-4a60-995b-557c7ed41a65-kube-api-access-mlt8l\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067232    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-config\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067273    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067304    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067327    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067350    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067388    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cccc1506-3c47-4a60-995b-557c7ed41a65-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067436    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067473    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067496    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-web-config\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067526    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.067711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067563    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.068033 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067737    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.068033 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067774    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.068033 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.067880    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.068439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.068221    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.068766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.068741    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.069055 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.069033    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.069859 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.069834    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.070647 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.070265    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.070647 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.070584    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.070647 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.070617    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.071133 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.071114    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cccc1506-3c47-4a60-995b-557c7ed41a65-config-out\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.071428 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.071402    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-config\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.071553 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.071528    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/cccc1506-3c47-4a60-995b-557c7ed41a65-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.072063 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.072031    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.072436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.072414    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-web-config\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.072915 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.072888    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cccc1506-3c47-4a60-995b-557c7ed41a65-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.073191 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.073166    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.073249 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.073220    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.073321 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.073306    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/cccc1506-3c47-4a60-995b-557c7ed41a65-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.121793 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.121756    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlt8l\" (UniqueName: \"kubernetes.io/projected/cccc1506-3c47-4a60-995b-557c7ed41a65-kube-api-access-mlt8l\") pod \"prometheus-k8s-0\" (UID: \"cccc1506-3c47-4a60-995b-557c7ed41a65\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.192312 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.192239    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2aa78254-1892-4d5f-83f3-77abc7a5e39c" path="/var/lib/kubelet/pods/2aa78254-1892-4d5f-83f3-77abc7a5e39c/volumes"
Nov 01 12:23:48.332808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.332774    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:23:48.645698 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.645667    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp" event={"ID":"4289e74e-8746-4732-9ffd-70b4e7c746c4","Type":"ContainerStarted","Data":"35991105b98c663ad292140b207177b3b940cac6b1c5190ef72ed1bba5965197"}
Nov 01 12:23:48.645962 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.645734    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp" event={"ID":"4289e74e-8746-4732-9ffd-70b4e7c746c4","Type":"ContainerStarted","Data":"5181c537a9401e0b028a5909af6f37f9041b0b821598341dbebd4f5923ccb97c"}
Nov 01 12:23:48.864765 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:48.864743    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 01 12:23:48.866311 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:23:48.866285    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcccc1506_3c47_4a60_995b_557c7ed41a65.slice/crio-fc5a03e84bafcbaa05f5a8f5110235e083bf706d59723a27f856cb4483a15a88 WatchSource:0}: Error finding container fc5a03e84bafcbaa05f5a8f5110235e083bf706d59723a27f856cb4483a15a88: Status 404 returned error can't find the container with id fc5a03e84bafcbaa05f5a8f5110235e083bf706d59723a27f856cb4483a15a88
Nov 01 12:23:49.112404 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:23:49.112371    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:23:49.651711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:49.651678    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp" event={"ID":"4289e74e-8746-4732-9ffd-70b4e7c746c4","Type":"ContainerStarted","Data":"960c0d36bdca688475f640fc6ae0caf0baf7f41428b0b4ba3648d8c6cb0d0a96"}
Nov 01 12:23:49.653245 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:49.653220    4095 generic.go:356] "Generic (PLEG): container finished" podID="cccc1506-3c47-4a60-995b-557c7ed41a65" containerID="a69658985d6c188543118d323261d104e42c33df0de9f5bacf00bd782b5fa216" exitCode=0
Nov 01 12:23:49.653389 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:49.653261    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cccc1506-3c47-4a60-995b-557c7ed41a65","Type":"ContainerDied","Data":"a69658985d6c188543118d323261d104e42c33df0de9f5bacf00bd782b5fa216"}
Nov 01 12:23:49.653389 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:49.653303    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cccc1506-3c47-4a60-995b-557c7ed41a65","Type":"ContainerStarted","Data":"fc5a03e84bafcbaa05f5a8f5110235e083bf706d59723a27f856cb4483a15a88"}
Nov 01 12:23:49.766713 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:49.766672    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c749d9f6b-86ntp" podStartSLOduration=3.065856546 podStartE2EDuration="5.766658006s" totalImagesPullingTime="2.70080146s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:23:49.765781147 +0000 UTC m=+89.683576813" watchObservedRunningTime="2026-11-01 12:23:49.766658006 +0000 UTC m=+89.684453672"
Nov 01 12:23:50.661475 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:50.661431    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cccc1506-3c47-4a60-995b-557c7ed41a65","Type":"ContainerStarted","Data":"000d7e7a09ea0d3a39ab402cd48ef39e85cc39f8bd534640eccfc41b74e4686e"}
Nov 01 12:23:50.661848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:50.661506    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cccc1506-3c47-4a60-995b-557c7ed41a65","Type":"ContainerStarted","Data":"02fec291d5a59a6a26049d4dcefe9a30dda5bac9816fae23c4928d3b935a318d"}
Nov 01 12:23:50.661848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:50.661537    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cccc1506-3c47-4a60-995b-557c7ed41a65","Type":"ContainerStarted","Data":"f5dce7a505f3d30947954754a876d9a6ce47744a150d19989a27fd8934925055"}
Nov 01 12:23:50.661848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:50.661575    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cccc1506-3c47-4a60-995b-557c7ed41a65","Type":"ContainerStarted","Data":"6650e5a094e9df7406d1cf2beb40d5271289d4a5fe7ff2036ad9eee444a9d58e"}
Nov 01 12:23:50.661848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:50.661605    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cccc1506-3c47-4a60-995b-557c7ed41a65","Type":"ContainerStarted","Data":"eb90cc84dcdc18c09a6cab56dc10f1929d9bb3867bb3fb87d6e68de9cf16298e"}
Nov 01 12:23:50.661848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:50.661626    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"cccc1506-3c47-4a60-995b-557c7ed41a65","Type":"ContainerStarted","Data":"16817a759a72608ca92e848ca658a221cb2f5aa702ba37f272da6b4456de559a"}
Nov 01 12:23:50.780929 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:50.780877    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.7658489729999998 podStartE2EDuration="3.780864563s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.01559ms" isStatefulPod=false podCreationTimestamp="2026-11-01 12:23:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:23:50.779025235 +0000 UTC m=+90.696820902" watchObservedRunningTime="2026-11-01 12:23:50.780864563 +0000 UTC m=+90.698660228"
Nov 01 12:23:53.333020 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:23:53.332989    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:24:19.111354 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:24:19.111317    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:24:48.333483 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:24:48.333403    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:24:48.348951 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:24:48.348924    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:24:48.885440 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:24:48.885415    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 01 12:24:49.112605 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:24:49.112569    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:25:19.112024 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:25:19.111985    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:25:49.111547 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:25:49.111507    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:26:19.113919 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:26:19.113874    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:26:49.112043 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:26:49.112008    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:27:19.112213 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:27:19.112177    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:27:20.190741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:27:20.190717    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:27:20.191219 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:27:20.191007    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:27:49.112485 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:27:49.112404    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:28:19.111689 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:28:19.111656    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:28:19.210628 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.210597    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"]
Nov 01 12:28:19.213939 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.213916    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"
Nov 01 12:28:19.271868 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.271844    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5vwt\" (UniqueName: \"kubernetes.io/projected/8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4-kube-api-access-d5vwt\") pod \"jobset-operator-674cd59dc5-rf7k6\" (UID: \"8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"
Nov 01 12:28:19.272016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.271895    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4-tmp\") pod \"jobset-operator-674cd59dc5-rf7k6\" (UID: \"8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"
Nov 01 12:28:19.314833 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.314803    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 01 12:28:19.315023 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.314910    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-fvdn6\""
Nov 01 12:28:19.315023 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.314988    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 01 12:28:19.315807 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.315790    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"]
Nov 01 12:28:19.372637 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.372573    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5vwt\" (UniqueName: \"kubernetes.io/projected/8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4-kube-api-access-d5vwt\") pod \"jobset-operator-674cd59dc5-rf7k6\" (UID: \"8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"
Nov 01 12:28:19.372637 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.372606    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4-tmp\") pod \"jobset-operator-674cd59dc5-rf7k6\" (UID: \"8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"
Nov 01 12:28:19.372940 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.372926    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4-tmp\") pod \"jobset-operator-674cd59dc5-rf7k6\" (UID: \"8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"
Nov 01 12:28:19.426843 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.426814    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5vwt\" (UniqueName: \"kubernetes.io/projected/8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4-kube-api-access-d5vwt\") pod \"jobset-operator-674cd59dc5-rf7k6\" (UID: \"8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"
Nov 01 12:28:19.540379 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.540354    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"
Nov 01 12:28:19.898648 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.898625    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6"]
Nov 01 12:28:19.900644 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:28:19.900618    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a25873c_0e84_4ae3_8fda_1e5bf64ce9e4.slice/crio-0125dc6e893bb12ecadcff64d4c218ca146587699ec03ed2e685c0e49fd415be WatchSource:0}: Error finding container 0125dc6e893bb12ecadcff64d4c218ca146587699ec03ed2e685c0e49fd415be: Status 404 returned error can't find the container with id 0125dc6e893bb12ecadcff64d4c218ca146587699ec03ed2e685c0e49fd415be
Nov 01 12:28:19.902536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:19.902518    4095 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:28:20.655016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:20.654959    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6" event={"ID":"8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4","Type":"ContainerStarted","Data":"0125dc6e893bb12ecadcff64d4c218ca146587699ec03ed2e685c0e49fd415be"}
Nov 01 12:28:24.672885 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:24.672844    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6" event={"ID":"8a25873c-0e84-4ae3-8fda-1e5bf64ce9e4","Type":"ContainerStarted","Data":"b4e04623e15731e2b70db1ec54aac3641d556a7add8fb43258fc535e39da691d"}
Nov 01 12:28:24.781430 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:24.781377    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-rf7k6" podStartSLOduration=1.506116831 podStartE2EDuration="5.781363943s" totalImagesPullingTime="4.275247112s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:24.780671552 +0000 UTC m=+364.698467218" watchObservedRunningTime="2026-11-01 12:28:24.781363943 +0000 UTC m=+364.699159609"
Nov 01 12:28:30.681603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.681562    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"]
Nov 01 12:28:30.685429 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.685405    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.778560 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.778525    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e2ead30-67f2-4b95-8dc4-d928926d2553-tmp\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.778717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.778582    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e2ead30-67f2-4b95-8dc4-d928926d2553-metrics-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.778717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.778687    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4jct\" (UniqueName: \"kubernetes.io/projected/6e2ead30-67f2-4b95-8dc4-d928926d2553-kube-api-access-k4jct\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.778812 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.778725    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e2ead30-67f2-4b95-8dc4-d928926d2553-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.778812 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.778741    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e2ead30-67f2-4b95-8dc4-d928926d2553-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.778895 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.778830    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e2ead30-67f2-4b95-8dc4-d928926d2553-manifests\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.789225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.789204    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 01 12:28:30.789225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.789206    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 01 12:28:30.789348 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.789229    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 01 12:28:30.789348 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.789241    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 01 12:28:30.789348 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.789283    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hqxjc\""
Nov 01 12:28:30.789446 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.789365    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 01 12:28:30.816026 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.816004    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"]
Nov 01 12:28:30.879826 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.879804    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e2ead30-67f2-4b95-8dc4-d928926d2553-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.879955 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.879829    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e2ead30-67f2-4b95-8dc4-d928926d2553-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.879955 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.879858    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e2ead30-67f2-4b95-8dc4-d928926d2553-manifests\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.879955 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.879881    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e2ead30-67f2-4b95-8dc4-d928926d2553-tmp\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.879955 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.879920    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e2ead30-67f2-4b95-8dc4-d928926d2553-metrics-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.880170 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.879958    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4jct\" (UniqueName: \"kubernetes.io/projected/6e2ead30-67f2-4b95-8dc4-d928926d2553-kube-api-access-k4jct\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.880256 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.880238    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6e2ead30-67f2-4b95-8dc4-d928926d2553-manifests\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.882213 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.882184    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e2ead30-67f2-4b95-8dc4-d928926d2553-tmp\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.882414 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.882397    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6e2ead30-67f2-4b95-8dc4-d928926d2553-webhook-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.882478 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.882416    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6e2ead30-67f2-4b95-8dc4-d928926d2553-metrics-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.882478 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.882446    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6e2ead30-67f2-4b95-8dc4-d928926d2553-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:30.934523 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:30.934456    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4jct\" (UniqueName: \"kubernetes.io/projected/6e2ead30-67f2-4b95-8dc4-d928926d2553-kube-api-access-k4jct\") pod \"opendatahub-operator-controller-manager-fcfb9fcfb-brd47\" (UID: \"6e2ead30-67f2-4b95-8dc4-d928926d2553\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:31.000243 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:31.000204    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:31.465957 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:31.465933    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"]
Nov 01 12:28:31.467965 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:28:31.467938    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e2ead30_67f2_4b95_8dc4_d928926d2553.slice/crio-9fd5c42d252ef52d9cb5380e22b3cc866663d6bbf66ed7d271c519be47b75a11 WatchSource:0}: Error finding container 9fd5c42d252ef52d9cb5380e22b3cc866663d6bbf66ed7d271c519be47b75a11: Status 404 returned error can't find the container with id 9fd5c42d252ef52d9cb5380e22b3cc866663d6bbf66ed7d271c519be47b75a11
Nov 01 12:28:31.699482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:31.699445    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47" event={"ID":"6e2ead30-67f2-4b95-8dc4-d928926d2553","Type":"ContainerStarted","Data":"9fd5c42d252ef52d9cb5380e22b3cc866663d6bbf66ed7d271c519be47b75a11"}
Nov 01 12:28:37.728233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:37.728201    4095 generic.go:356] "Generic (PLEG): container finished" podID="6e2ead30-67f2-4b95-8dc4-d928926d2553" containerID="47a10d45e9e79a7d8285d5aedc09b378b2f86c04d13b742d47379e753a7bbf64" exitCode=0
Nov 01 12:28:37.728608 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:37.728273    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47" event={"ID":"6e2ead30-67f2-4b95-8dc4-d928926d2553","Type":"ContainerDied","Data":"47a10d45e9e79a7d8285d5aedc09b378b2f86c04d13b742d47379e753a7bbf64"}
Nov 01 12:28:38.734567 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:38.734528    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47" event={"ID":"6e2ead30-67f2-4b95-8dc4-d928926d2553","Type":"ContainerStarted","Data":"d48188e82b76c10bcf28f618603d651aa20a6f29d02ab2f0a3a2b96c44b59fdb"}
Nov 01 12:28:38.734946 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:38.734654    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:38.866495 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:38.866400    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47" podStartSLOduration=1.909125048 podStartE2EDuration="8.866386895s" totalImagesPullingTime="6.472238048s" totalInitContainerRuntime="485.023799ms" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:30 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:38.8643506 +0000 UTC m=+378.782146266" watchObservedRunningTime="2026-11-01 12:28:38.866386895 +0000 UTC m=+378.784182560"
Nov 01 12:28:49.112299 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:28:49.112256    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:28:49.741175 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:49.741147    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47"
Nov 01 12:28:49.896622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:49.896595    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"]
Nov 01 12:28:49.904361 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:49.904342    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:28:49.935146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:49.935121    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"]
Nov 01 12:28:49.954336 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:49.954315    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 01 12:28:49.954460 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:49.954363    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 01 12:28:50.019402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.019314    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-ptk44\""
Nov 01 12:28:50.055643 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.055610    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ae7e690-ba05-45ab-9b7f-ea875d2e526a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sskbf\" (UID: \"7ae7e690-ba05-45ab-9b7f-ea875d2e526a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:28:50.055815 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.055654    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvhrq\" (UniqueName: \"kubernetes.io/projected/7ae7e690-ba05-45ab-9b7f-ea875d2e526a-kube-api-access-tvhrq\") pod \"cert-manager-webhook-5dbbbf75cc-sskbf\" (UID: \"7ae7e690-ba05-45ab-9b7f-ea875d2e526a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:28:50.156534 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.156501    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvhrq\" (UniqueName: \"kubernetes.io/projected/7ae7e690-ba05-45ab-9b7f-ea875d2e526a-kube-api-access-tvhrq\") pod \"cert-manager-webhook-5dbbbf75cc-sskbf\" (UID: \"7ae7e690-ba05-45ab-9b7f-ea875d2e526a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:28:50.156952 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.156582    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ae7e690-ba05-45ab-9b7f-ea875d2e526a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sskbf\" (UID: \"7ae7e690-ba05-45ab-9b7f-ea875d2e526a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:28:50.211545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.211517    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/7ae7e690-ba05-45ab-9b7f-ea875d2e526a-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-sskbf\" (UID: \"7ae7e690-ba05-45ab-9b7f-ea875d2e526a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:28:50.211687 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.211575    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvhrq\" (UniqueName: \"kubernetes.io/projected/7ae7e690-ba05-45ab-9b7f-ea875d2e526a-kube-api-access-tvhrq\") pod \"cert-manager-webhook-5dbbbf75cc-sskbf\" (UID: \"7ae7e690-ba05-45ab-9b7f-ea875d2e526a\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:28:50.215777 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.215756    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:28:50.575775 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.575752    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"]
Nov 01 12:28:50.577716 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:28:50.577688    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7ae7e690_ba05_45ab_9b7f_ea875d2e526a.slice/crio-67e5c7313aa31625b9a0d0426020f60c2f36e21f0077ef9751b58daf1b7d4bdd WatchSource:0}: Error finding container 67e5c7313aa31625b9a0d0426020f60c2f36e21f0077ef9751b58daf1b7d4bdd: Status 404 returned error can't find the container with id 67e5c7313aa31625b9a0d0426020f60c2f36e21f0077ef9751b58daf1b7d4bdd
Nov 01 12:28:50.783087 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:50.783054    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf" event={"ID":"7ae7e690-ba05-45ab-9b7f-ea875d2e526a","Type":"ContainerStarted","Data":"67e5c7313aa31625b9a0d0426020f60c2f36e21f0077ef9751b58daf1b7d4bdd"}
Nov 01 12:28:52.209912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.209884    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"]
Nov 01 12:28:52.285308 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.285281    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"]
Nov 01 12:28:52.285461 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.285369    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"
Nov 01 12:28:52.376640 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.376617    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-h7n6q\" (UID: \"9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"
Nov 01 12:28:52.376773 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.376665    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxhtl\" (UniqueName: \"kubernetes.io/projected/9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06-kube-api-access-fxhtl\") pod \"cert-manager-cainjector-5c57bbd4fc-h7n6q\" (UID: \"9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"
Nov 01 12:28:52.385785 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.385763    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-8qfj8\""
Nov 01 12:28:52.478148 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.478079    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-h7n6q\" (UID: \"9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"
Nov 01 12:28:52.478290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.478157    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxhtl\" (UniqueName: \"kubernetes.io/projected/9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06-kube-api-access-fxhtl\") pod \"cert-manager-cainjector-5c57bbd4fc-h7n6q\" (UID: \"9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"
Nov 01 12:28:52.533396 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.533368    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-h7n6q\" (UID: \"9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"
Nov 01 12:28:52.533560 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.533458    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxhtl\" (UniqueName: \"kubernetes.io/projected/9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06-kube-api-access-fxhtl\") pod \"cert-manager-cainjector-5c57bbd4fc-h7n6q\" (UID: \"9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"
Nov 01 12:28:52.611041 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:52.611008    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"
Nov 01 12:28:53.010908 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:53.010855    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q"]
Nov 01 12:28:53.015701 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:28:53.015666    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9e6f2a9f_ac34_40c5_9a2b_bd60c6430e06.slice/crio-0a2cce58af1dd09994d138fd868c59b3e78c741e232706bc6fd65d731ac5187e WatchSource:0}: Error finding container 0a2cce58af1dd09994d138fd868c59b3e78c741e232706bc6fd65d731ac5187e: Status 404 returned error can't find the container with id 0a2cce58af1dd09994d138fd868c59b3e78c741e232706bc6fd65d731ac5187e
Nov 01 12:28:53.799230 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:53.799184    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q" event={"ID":"9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06","Type":"ContainerStarted","Data":"0a2cce58af1dd09994d138fd868c59b3e78c741e232706bc6fd65d731ac5187e"}
Nov 01 12:28:55.810313 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:55.810267    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf" event={"ID":"7ae7e690-ba05-45ab-9b7f-ea875d2e526a","Type":"ContainerStarted","Data":"308ba7a1a78fc0365237c093a936ffce7914f62055849a09ea151d7c735c716c"}
Nov 01 12:28:55.810841 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:55.810401    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:28:55.918521 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:55.918465    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf" podStartSLOduration=2.515635564 podStartE2EDuration="6.918451911s" totalImagesPullingTime="4.402816347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:55.917345801 +0000 UTC m=+395.835141468" watchObservedRunningTime="2026-11-01 12:28:55.918451911 +0000 UTC m=+395.836247576"
Nov 01 12:28:56.816005 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:56.815959    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q" event={"ID":"9e6f2a9f-ac34-40c5-9a2b-bd60c6430e06","Type":"ContainerStarted","Data":"357ae27af9d307e11fcd6ae6f504d31a3ae95d3b4c7b7c0afa2fb2b8c53d1077"}
Nov 01 12:28:56.925507 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:56.925444    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-h7n6q" podStartSLOduration=1.900728173 podStartE2EDuration="4.925425687s" totalImagesPullingTime="3.024697514s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:28:56.924307925 +0000 UTC m=+396.842103618" watchObservedRunningTime="2026-11-01 12:28:56.925425687 +0000 UTC m=+396.843221355"
Nov 01 12:28:58.427699 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.427668    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"]
Nov 01 12:28:58.431058 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.431043    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.483266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.483240    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 01 12:28:58.484985 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.484948    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"]
Nov 01 12:28:58.530408 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.530383    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 01 12:28:58.530546 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.530415    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-ckzhr\""
Nov 01 12:28:58.530599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.530557    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 01 12:28:58.530599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.530591    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 01 12:28:58.540309 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.540280    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8bf1a5f6-441b-45cd-a923-74d51fcaa90b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nhgx4\" (UID: \"8bf1a5f6-441b-45cd-a923-74d51fcaa90b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.540445 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.540424    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5l5qb\" (UniqueName: \"kubernetes.io/projected/8bf1a5f6-441b-45cd-a923-74d51fcaa90b-kube-api-access-5l5qb\") pod \"kuadrant-console-plugin-66c4fb575d-nhgx4\" (UID: \"8bf1a5f6-441b-45cd-a923-74d51fcaa90b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.540523 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.540506    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bf1a5f6-441b-45cd-a923-74d51fcaa90b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nhgx4\" (UID: \"8bf1a5f6-441b-45cd-a923-74d51fcaa90b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.641704 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.641676    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5l5qb\" (UniqueName: \"kubernetes.io/projected/8bf1a5f6-441b-45cd-a923-74d51fcaa90b-kube-api-access-5l5qb\") pod \"kuadrant-console-plugin-66c4fb575d-nhgx4\" (UID: \"8bf1a5f6-441b-45cd-a923-74d51fcaa90b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.641896 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.641722    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bf1a5f6-441b-45cd-a923-74d51fcaa90b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nhgx4\" (UID: \"8bf1a5f6-441b-45cd-a923-74d51fcaa90b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.641896 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.641821    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8bf1a5f6-441b-45cd-a923-74d51fcaa90b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nhgx4\" (UID: \"8bf1a5f6-441b-45cd-a923-74d51fcaa90b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.642484 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.642466    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8bf1a5f6-441b-45cd-a923-74d51fcaa90b-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-nhgx4\" (UID: \"8bf1a5f6-441b-45cd-a923-74d51fcaa90b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.644292 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.644270    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8bf1a5f6-441b-45cd-a923-74d51fcaa90b-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-nhgx4\" (UID: \"8bf1a5f6-441b-45cd-a923-74d51fcaa90b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.696470 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.696403    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5l5qb\" (UniqueName: \"kubernetes.io/projected/8bf1a5f6-441b-45cd-a923-74d51fcaa90b-kube-api-access-5l5qb\") pod \"kuadrant-console-plugin-66c4fb575d-nhgx4\" (UID: \"8bf1a5f6-441b-45cd-a923-74d51fcaa90b\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:58.741461 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:58.741422    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"
Nov 01 12:28:59.101098 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:59.101075    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4"]
Nov 01 12:28:59.103300 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:28:59.103265    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bf1a5f6_441b_45cd_a923_74d51fcaa90b.slice/crio-dbb446b54cf053b8af082e37518065418f83d40f514ea16b34d170135999c29b WatchSource:0}: Error finding container dbb446b54cf053b8af082e37518065418f83d40f514ea16b34d170135999c29b: Status 404 returned error can't find the container with id dbb446b54cf053b8af082e37518065418f83d40f514ea16b34d170135999c29b
Nov 01 12:28:59.829621 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:28:59.829586    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4" event={"ID":"8bf1a5f6-441b-45cd-a923-74d51fcaa90b","Type":"ContainerStarted","Data":"dbb446b54cf053b8af082e37518065418f83d40f514ea16b34d170135999c29b"}
Nov 01 12:29:00.944022 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:00.943935    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-8nt9w"]
Nov 01 12:29:00.947196 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:00.947176    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8nt9w"
Nov 01 12:29:01.044713 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.044683    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lpdzq\""
Nov 01 12:29:01.046167 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.046144    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8nt9w"]
Nov 01 12:29:01.063387 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.063363    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6411969c-a482-41ca-b605-9864c7a730d0-bound-sa-token\") pod \"cert-manager-597b95cb84-8nt9w\" (UID: \"6411969c-a482-41ca-b605-9864c7a730d0\") " pod="cert-manager/cert-manager-597b95cb84-8nt9w"
Nov 01 12:29:01.063497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.063414    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llwjj\" (UniqueName: \"kubernetes.io/projected/6411969c-a482-41ca-b605-9864c7a730d0-kube-api-access-llwjj\") pod \"cert-manager-597b95cb84-8nt9w\" (UID: \"6411969c-a482-41ca-b605-9864c7a730d0\") " pod="cert-manager/cert-manager-597b95cb84-8nt9w"
Nov 01 12:29:01.164922 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.164854    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6411969c-a482-41ca-b605-9864c7a730d0-bound-sa-token\") pod \"cert-manager-597b95cb84-8nt9w\" (UID: \"6411969c-a482-41ca-b605-9864c7a730d0\") " pod="cert-manager/cert-manager-597b95cb84-8nt9w"
Nov 01 12:29:01.165193 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.164949    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llwjj\" (UniqueName: \"kubernetes.io/projected/6411969c-a482-41ca-b605-9864c7a730d0-kube-api-access-llwjj\") pod \"cert-manager-597b95cb84-8nt9w\" (UID: \"6411969c-a482-41ca-b605-9864c7a730d0\") " pod="cert-manager/cert-manager-597b95cb84-8nt9w"
Nov 01 12:29:01.221189 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.221085    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llwjj\" (UniqueName: \"kubernetes.io/projected/6411969c-a482-41ca-b605-9864c7a730d0-kube-api-access-llwjj\") pod \"cert-manager-597b95cb84-8nt9w\" (UID: \"6411969c-a482-41ca-b605-9864c7a730d0\") " pod="cert-manager/cert-manager-597b95cb84-8nt9w"
Nov 01 12:29:01.223320 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.223271    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6411969c-a482-41ca-b605-9864c7a730d0-bound-sa-token\") pod \"cert-manager-597b95cb84-8nt9w\" (UID: \"6411969c-a482-41ca-b605-9864c7a730d0\") " pod="cert-manager/cert-manager-597b95cb84-8nt9w"
Nov 01 12:29:01.259388 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.259117    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-8nt9w"
Nov 01 12:29:01.645136 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.645113    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-8nt9w"]
Nov 01 12:29:01.647559 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:29:01.647533    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6411969c_a482_41ca_b605_9864c7a730d0.slice/crio-298fa6b7bc802453cfd38968c5e8c4ba30d6b62a0e2e1797302a63214bbd7323 WatchSource:0}: Error finding container 298fa6b7bc802453cfd38968c5e8c4ba30d6b62a0e2e1797302a63214bbd7323: Status 404 returned error can't find the container with id 298fa6b7bc802453cfd38968c5e8c4ba30d6b62a0e2e1797302a63214bbd7323
Nov 01 12:29:01.818700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.818660    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf"
Nov 01 12:29:01.841436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.841404    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8nt9w" event={"ID":"6411969c-a482-41ca-b605-9864c7a730d0","Type":"ContainerStarted","Data":"b4bf0f57cc86509077537effa15b0f8fb0280dc45b2447efaaaba214f74a1b56"}
Nov 01 12:29:01.841583 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:01.841444    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-8nt9w" event={"ID":"6411969c-a482-41ca-b605-9864c7a730d0","Type":"ContainerStarted","Data":"298fa6b7bc802453cfd38968c5e8c4ba30d6b62a0e2e1797302a63214bbd7323"}
Nov 01 12:29:02.035443 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:02.035341    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-8nt9w" podStartSLOduration=2.035325728 podStartE2EDuration="2.035325728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:02.033780957 +0000 UTC m=+401.951576623" watchObservedRunningTime="2026-11-01 12:29:02.035325728 +0000 UTC m=+401.953121388"
Nov 01 12:29:05.042601 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.042558    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"]
Nov 01 12:29:05.049819 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.049786    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.147287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.147253    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 01 12:29:05.147287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.147251    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 01 12:29:05.147525 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.147462    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 01 12:29:05.147525 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.147471    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 01 12:29:05.147525 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.147465    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-grtjg\""
Nov 01 12:29:05.149155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.149130    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"]
Nov 01 12:29:05.207479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.207444    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c54b04d-ee14-4f0c-a7ff-0cba080217a2-tmp\") pod \"kube-auth-proxy-7bf7dc98fb-4t6bt\" (UID: \"5c54b04d-ee14-4f0c-a7ff-0cba080217a2\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.207675 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.207608    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c54b04d-ee14-4f0c-a7ff-0cba080217a2-tls-certs\") pod \"kube-auth-proxy-7bf7dc98fb-4t6bt\" (UID: \"5c54b04d-ee14-4f0c-a7ff-0cba080217a2\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.207675 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.207650    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqdg2\" (UniqueName: \"kubernetes.io/projected/5c54b04d-ee14-4f0c-a7ff-0cba080217a2-kube-api-access-dqdg2\") pod \"kube-auth-proxy-7bf7dc98fb-4t6bt\" (UID: \"5c54b04d-ee14-4f0c-a7ff-0cba080217a2\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.309391 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.309308    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c54b04d-ee14-4f0c-a7ff-0cba080217a2-tmp\") pod \"kube-auth-proxy-7bf7dc98fb-4t6bt\" (UID: \"5c54b04d-ee14-4f0c-a7ff-0cba080217a2\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.309655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.309423    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c54b04d-ee14-4f0c-a7ff-0cba080217a2-tls-certs\") pod \"kube-auth-proxy-7bf7dc98fb-4t6bt\" (UID: \"5c54b04d-ee14-4f0c-a7ff-0cba080217a2\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.309655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.309455    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqdg2\" (UniqueName: \"kubernetes.io/projected/5c54b04d-ee14-4f0c-a7ff-0cba080217a2-kube-api-access-dqdg2\") pod \"kube-auth-proxy-7bf7dc98fb-4t6bt\" (UID: \"5c54b04d-ee14-4f0c-a7ff-0cba080217a2\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.312547 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.312515    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c54b04d-ee14-4f0c-a7ff-0cba080217a2-tmp\") pod \"kube-auth-proxy-7bf7dc98fb-4t6bt\" (UID: \"5c54b04d-ee14-4f0c-a7ff-0cba080217a2\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.312805 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.312786    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c54b04d-ee14-4f0c-a7ff-0cba080217a2-tls-certs\") pod \"kube-auth-proxy-7bf7dc98fb-4t6bt\" (UID: \"5c54b04d-ee14-4f0c-a7ff-0cba080217a2\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.371275 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.371241    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqdg2\" (UniqueName: \"kubernetes.io/projected/5c54b04d-ee14-4f0c-a7ff-0cba080217a2-kube-api-access-dqdg2\") pod \"kube-auth-proxy-7bf7dc98fb-4t6bt\" (UID: \"5c54b04d-ee14-4f0c-a7ff-0cba080217a2\") " pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:05.664332 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:05.664300    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"
Nov 01 12:29:06.046638 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:06.046611    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt"]
Nov 01 12:29:06.048089 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:29:06.048065    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c54b04d_ee14_4f0c_a7ff_0cba080217a2.slice/crio-5a85dd599e3fc4d552f6b3308a21ad1cc3be98cad49409679a67a0a7de490cb6 WatchSource:0}: Error finding container 5a85dd599e3fc4d552f6b3308a21ad1cc3be98cad49409679a67a0a7de490cb6: Status 404 returned error can't find the container with id 5a85dd599e3fc4d552f6b3308a21ad1cc3be98cad49409679a67a0a7de490cb6
Nov 01 12:29:06.869819 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:06.869766    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt" event={"ID":"5c54b04d-ee14-4f0c-a7ff-0cba080217a2","Type":"ContainerStarted","Data":"5a85dd599e3fc4d552f6b3308a21ad1cc3be98cad49409679a67a0a7de490cb6"}
Nov 01 12:29:07.367458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.367426    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"]
Nov 01 12:29:07.371201 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.371175    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:07.423477 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.423422    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 01 12:29:07.426002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.425954    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"]
Nov 01 12:29:07.426353 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.426326    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/95f461ff-1d76-404b-a23b-680735e53700-config-file\") pod \"limitador-limitador-74dfcfb894-d6kdz\" (UID: \"95f461ff-1d76-404b-a23b-680735e53700\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:07.426456 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.426432    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8whtq\" (UniqueName: \"kubernetes.io/projected/95f461ff-1d76-404b-a23b-680735e53700-kube-api-access-8whtq\") pod \"limitador-limitador-74dfcfb894-d6kdz\" (UID: \"95f461ff-1d76-404b-a23b-680735e53700\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:07.527597 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.527566    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8whtq\" (UniqueName: \"kubernetes.io/projected/95f461ff-1d76-404b-a23b-680735e53700-kube-api-access-8whtq\") pod \"limitador-limitador-74dfcfb894-d6kdz\" (UID: \"95f461ff-1d76-404b-a23b-680735e53700\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:07.527777 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.527647    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/95f461ff-1d76-404b-a23b-680735e53700-config-file\") pod \"limitador-limitador-74dfcfb894-d6kdz\" (UID: \"95f461ff-1d76-404b-a23b-680735e53700\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:07.528498 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.528477    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/95f461ff-1d76-404b-a23b-680735e53700-config-file\") pod \"limitador-limitador-74dfcfb894-d6kdz\" (UID: \"95f461ff-1d76-404b-a23b-680735e53700\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:07.592519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.588035    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8whtq\" (UniqueName: \"kubernetes.io/projected/95f461ff-1d76-404b-a23b-680735e53700-kube-api-access-8whtq\") pod \"limitador-limitador-74dfcfb894-d6kdz\" (UID: \"95f461ff-1d76-404b-a23b-680735e53700\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:07.603019 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.602761    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"]
Nov 01 12:29:07.687206 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.687124    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:07.877544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.877510    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4" event={"ID":"8bf1a5f6-441b-45cd-a923-74d51fcaa90b","Type":"ContainerStarted","Data":"9343ec91c64b94abcb4b5b14d7c2a632f2548c4132e1e716dfbab899ae642fd4"}
Nov 01 12:29:07.986252 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:07.986161    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-nhgx4" podStartSLOduration=1.677246041 podStartE2EDuration="9.986146553s" totalImagesPullingTime="8.308900512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:28:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:07.985328895 +0000 UTC m=+407.903124563" watchObservedRunningTime="2026-11-01 12:29:07.986146553 +0000 UTC m=+407.903942219"
Nov 01 12:29:08.090930 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:08.090899    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"]
Nov 01 12:29:08.093119 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:29:08.093092    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f461ff_1d76_404b_a23b_680735e53700.slice/crio-85d3be377d02055f7ec237bbaba3dd701910c9d9ca690a7af258ef3e3d0f9813 WatchSource:0}: Error finding container 85d3be377d02055f7ec237bbaba3dd701910c9d9ca690a7af258ef3e3d0f9813: Status 404 returned error can't find the container with id 85d3be377d02055f7ec237bbaba3dd701910c9d9ca690a7af258ef3e3d0f9813
Nov 01 12:29:08.883699 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:08.883653    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz" event={"ID":"95f461ff-1d76-404b-a23b-680735e53700","Type":"ContainerStarted","Data":"85d3be377d02055f7ec237bbaba3dd701910c9d9ca690a7af258ef3e3d0f9813"}
Nov 01 12:29:09.893054 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:09.893019    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt" event={"ID":"5c54b04d-ee14-4f0c-a7ff-0cba080217a2","Type":"ContainerStarted","Data":"4d65a60403ac2c8a7a6bf05bd5d09246f105225bd34a634e57992d4f088ac172"}
Nov 01 12:29:10.002297 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:10.002234    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7bf7dc98fb-4t6bt" podStartSLOduration=1.85020681 podStartE2EDuration="5.002217395s" totalImagesPullingTime="3.152010585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:10.001776858 +0000 UTC m=+409.919572523" watchObservedRunningTime="2026-11-01 12:29:10.002217395 +0000 UTC m=+409.920013060"
Nov 01 12:29:11.903105 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:11.903006    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz" event={"ID":"95f461ff-1d76-404b-a23b-680735e53700","Type":"ContainerStarted","Data":"8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f"}
Nov 01 12:29:11.903105 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:11.903073    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:12.011168 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:12.011119    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz" podStartSLOduration=1.491483733 podStartE2EDuration="5.011104067s" totalImagesPullingTime="3.519620334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:12.010508375 +0000 UTC m=+411.928304041" watchObservedRunningTime="2026-11-01 12:29:12.011104067 +0000 UTC m=+411.928899734"
Nov 01 12:29:19.113144 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:29:19.113098    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:29:22.907874 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:22.907837    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:26.945219 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:26.945183    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"]
Nov 01 12:29:26.945711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:26.945430    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz" podUID="95f461ff-1d76-404b-a23b-680735e53700" containerName="limitador" containerID="cri-o://8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f" gracePeriod=30
Nov 01 12:29:27.488127 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.488104    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:27.618350 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.618324    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95f461ff-1d76-404b-a23b-680735e53700-config-file\" (UniqueName: \"kubernetes.io/configmap/95f461ff-1d76-404b-a23b-680735e53700-config-file\") pod \"95f461ff-1d76-404b-a23b-680735e53700\" (UID: \"95f461ff-1d76-404b-a23b-680735e53700\") "
Nov 01 12:29:27.618528 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.618392    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95f461ff-1d76-404b-a23b-680735e53700-kube-api-access-8whtq\" (UniqueName: \"kubernetes.io/projected/95f461ff-1d76-404b-a23b-680735e53700-kube-api-access-8whtq\") pod \"95f461ff-1d76-404b-a23b-680735e53700\" (UID: \"95f461ff-1d76-404b-a23b-680735e53700\") "
Nov 01 12:29:27.618731 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.618707    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f461ff-1d76-404b-a23b-680735e53700-config-file" pod "95f461ff-1d76-404b-a23b-680735e53700" (UID: "95f461ff-1d76-404b-a23b-680735e53700"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:29:27.620506 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.620482    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f461ff-1d76-404b-a23b-680735e53700-kube-api-access-8whtq" pod "95f461ff-1d76-404b-a23b-680735e53700" (UID: "95f461ff-1d76-404b-a23b-680735e53700"). InnerVolumeSpecName "kube-api-access-8whtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:29:27.719439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.719402    4095 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/95f461ff-1d76-404b-a23b-680735e53700-config-file\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:29:27.719439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.719432    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8whtq\" (UniqueName: \"kubernetes.io/projected/95f461ff-1d76-404b-a23b-680735e53700-kube-api-access-8whtq\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:29:27.985130 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.985043    4095 generic.go:356] "Generic (PLEG): container finished" podID="95f461ff-1d76-404b-a23b-680735e53700" containerID="8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f" exitCode=0
Nov 01 12:29:27.985130 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.985117    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz" event={"ID":"95f461ff-1d76-404b-a23b-680735e53700","Type":"ContainerDied","Data":"8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f"}
Nov 01 12:29:27.985647 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.985142    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"
Nov 01 12:29:27.985647 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.985170    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-d6kdz" event={"ID":"95f461ff-1d76-404b-a23b-680735e53700","Type":"ContainerDied","Data":"85d3be377d02055f7ec237bbaba3dd701910c9d9ca690a7af258ef3e3d0f9813"}
Nov 01 12:29:27.985647 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.985199    4095 scope.go:122] "RemoveContainer" containerID="8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f"
Nov 01 12:29:27.994305 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.994287    4095 scope.go:122] "RemoveContainer" containerID="8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f"
Nov 01 12:29:27.994560 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:29:27.994541    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f\": container with ID starting with 8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f not found: ID does not exist" containerID="8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f"
Nov 01 12:29:27.994602 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:27.994571    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f"} err="failed to get container status \"8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f\": rpc error: code = NotFound desc = could not find container \"8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f\": container with ID starting with 8937669ba5849cb035768a5e1fa8e08d5e9ca9d7a8c6176b1dff1543cfe1639f not found: ID does not exist"
Nov 01 12:29:28.139943 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.139911    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"]
Nov 01 12:29:28.187245 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.187216    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-d6kdz"]
Nov 01 12:29:28.333202 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.333132    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"]
Nov 01 12:29:28.333538 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.333522    4095 cpu_manager.go:414] "containerMap: removing container" podUID="95f461ff-1d76-404b-a23b-680735e53700" containerName="limitador"
Nov 01 12:29:28.333619 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.333537    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95f461ff-1d76-404b-a23b-680735e53700" containerName="limitador"
Nov 01 12:29:28.333658 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.333629    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="95f461ff-1d76-404b-a23b-680735e53700" containerName="limitador"
Nov 01 12:29:28.338732 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.338711    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.392185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.392158    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-f66pl\""
Nov 01 12:29:28.392312 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.392255    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 01 12:29:28.392360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.392317    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 01 12:29:28.392952 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.392933    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"]
Nov 01 12:29:28.425723 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.425702    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6b9554e5-c0e4-411c-913b-aeab20226d3b-manager-config\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.425826 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.425766    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7klxm\" (UniqueName: \"kubernetes.io/projected/6b9554e5-c0e4-411c-913b-aeab20226d3b-kube-api-access-7klxm\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.425868 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.425830    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b9554e5-c0e4-411c-913b-aeab20226d3b-cert\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.425868 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.425853    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b9554e5-c0e4-411c-913b-aeab20226d3b-metrics-certs\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.465238 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.465220    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 01 12:29:28.526890 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.526865    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6b9554e5-c0e4-411c-913b-aeab20226d3b-manager-config\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.527069 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.526926    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7klxm\" (UniqueName: \"kubernetes.io/projected/6b9554e5-c0e4-411c-913b-aeab20226d3b-kube-api-access-7klxm\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.527069 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.526989    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b9554e5-c0e4-411c-913b-aeab20226d3b-cert\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.527069 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.527014    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b9554e5-c0e4-411c-913b-aeab20226d3b-metrics-certs\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.527564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.527544    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6b9554e5-c0e4-411c-913b-aeab20226d3b-manager-config\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.529633 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.529613    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b9554e5-c0e4-411c-913b-aeab20226d3b-cert\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.529698 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.529619    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6b9554e5-c0e4-411c-913b-aeab20226d3b-metrics-certs\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.581116 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.581093    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7klxm\" (UniqueName: \"kubernetes.io/projected/6b9554e5-c0e4-411c-913b-aeab20226d3b-kube-api-access-7klxm\") pod \"jobset-controller-manager-857ccf4696-frjrk\" (UID: \"6b9554e5-c0e4-411c-913b-aeab20226d3b\") " pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:28.651471 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:28.651441    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:29.041330 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:29:29.041300    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b9554e5_c0e4_411c_913b_aeab20226d3b.slice/crio-bf473c37fca6e7b3096a26b935dc6f2628b8bbbf9f1aee03632c47f69bb56ef5 WatchSource:0}: Error finding container bf473c37fca6e7b3096a26b935dc6f2628b8bbbf9f1aee03632c47f69bb56ef5: Status 404 returned error can't find the container with id bf473c37fca6e7b3096a26b935dc6f2628b8bbbf9f1aee03632c47f69bb56ef5
Nov 01 12:29:29.074274 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:29.074249    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"]
Nov 01 12:29:29.996452 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:29.996422    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk" event={"ID":"6b9554e5-c0e4-411c-913b-aeab20226d3b","Type":"ContainerStarted","Data":"bf473c37fca6e7b3096a26b935dc6f2628b8bbbf9f1aee03632c47f69bb56ef5"}
Nov 01 12:29:30.192344 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:30.188914    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95f461ff-1d76-404b-a23b-680735e53700" path="/var/lib/kubelet/pods/95f461ff-1d76-404b-a23b-680735e53700/volumes"
Nov 01 12:29:34.016331 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:34.016299    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk" event={"ID":"6b9554e5-c0e4-411c-913b-aeab20226d3b","Type":"ContainerStarted","Data":"8b4e840ee661ce65b32514b47a28302a66c91e60d484ce05f2184148762a425c"}
Nov 01 12:29:34.016730 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:34.016400    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:34.128363 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:34.128306    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk" podStartSLOduration=1.68564807 podStartE2EDuration="6.128292164s" totalImagesPullingTime="4.442644094s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:29:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:29:34.126667054 +0000 UTC m=+434.044462734" watchObservedRunningTime="2026-11-01 12:29:34.128292164 +0000 UTC m=+434.046087830"
Nov 01 12:29:45.026068 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:29:45.026039    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-857ccf4696-frjrk"
Nov 01 12:29:49.111535 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:29:49.111495    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:30:19.112518 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:30:19.112483    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:30:24.951432 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:24.951393    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-ktl4n"]
Nov 01 12:30:24.959754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:24.959727    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:25.008884 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.008846    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-ktl4n"]
Nov 01 12:30:25.029493 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.029460    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7pts\" (UniqueName: \"kubernetes.io/projected/649e29f1-f8a8-490e-a0ac-a729006388be-kube-api-access-n7pts\") pod \"model-serving-api-75bd89fdfc-ktl4n\" (UID: \"649e29f1-f8a8-490e-a0ac-a729006388be\") " pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:25.029647 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.029571    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649e29f1-f8a8-490e-a0ac-a729006388be-tls-certs\") pod \"model-serving-api-75bd89fdfc-ktl4n\" (UID: \"649e29f1-f8a8-490e-a0ac-a729006388be\") " pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:25.054571 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.054545    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 01 12:30:25.054571 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.054551    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 01 12:30:25.056766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.056744    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-kq855\""
Nov 01 12:30:25.056924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.056796    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Nov 01 12:30:25.130191 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.130160    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649e29f1-f8a8-490e-a0ac-a729006388be-tls-certs\") pod \"model-serving-api-75bd89fdfc-ktl4n\" (UID: \"649e29f1-f8a8-490e-a0ac-a729006388be\") " pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:25.130346 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.130238    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7pts\" (UniqueName: \"kubernetes.io/projected/649e29f1-f8a8-490e-a0ac-a729006388be-kube-api-access-n7pts\") pod \"model-serving-api-75bd89fdfc-ktl4n\" (UID: \"649e29f1-f8a8-490e-a0ac-a729006388be\") " pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:25.132580 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.132555    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/649e29f1-f8a8-490e-a0ac-a729006388be-tls-certs\") pod \"model-serving-api-75bd89fdfc-ktl4n\" (UID: \"649e29f1-f8a8-490e-a0ac-a729006388be\") " pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:25.186355 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.186328    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7pts\" (UniqueName: \"kubernetes.io/projected/649e29f1-f8a8-490e-a0ac-a729006388be-kube-api-access-n7pts\") pod \"model-serving-api-75bd89fdfc-ktl4n\" (UID: \"649e29f1-f8a8-490e-a0ac-a729006388be\") " pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:25.272414 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.272336    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:25.635700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:25.635673    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-ktl4n"]
Nov 01 12:30:25.637612 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:30:25.637586    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649e29f1_f8a8_490e_a0ac_a729006388be.slice/crio-a7efbf860a4d99fb3ae69ea473507c7ad9cb809b0262cf402af363d1689dbf42 WatchSource:0}: Error finding container a7efbf860a4d99fb3ae69ea473507c7ad9cb809b0262cf402af363d1689dbf42: Status 404 returned error can't find the container with id a7efbf860a4d99fb3ae69ea473507c7ad9cb809b0262cf402af363d1689dbf42
Nov 01 12:30:26.241284 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:26.241242    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n" event={"ID":"649e29f1-f8a8-490e-a0ac-a729006388be","Type":"ContainerStarted","Data":"a7efbf860a4d99fb3ae69ea473507c7ad9cb809b0262cf402af363d1689dbf42"}
Nov 01 12:30:28.251439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:28.251347    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n" event={"ID":"649e29f1-f8a8-490e-a0ac-a729006388be","Type":"ContainerStarted","Data":"b41e8330f97530b6842fa73511fcbd4f6eac5a250aa27e93c474da1afc498fbd"}
Nov 01 12:30:28.251439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:28.251409    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:28.359876 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:28.359821    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n" podStartSLOduration=2.023098273 podStartE2EDuration="4.359808239s" totalImagesPullingTime="2.336709966s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:28.359568648 +0000 UTC m=+488.277364315" watchObservedRunningTime="2026-11-01 12:30:28.359808239 +0000 UTC m=+488.277603905"
Nov 01 12:30:39.260279 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:39.260203    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-ktl4n"
Nov 01 12:30:42.051379 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.051347    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"]
Nov 01 12:30:42.054558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.054543    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.154760 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.154726    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 01 12:30:42.154760 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.154726    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 01 12:30:42.155034 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.154778    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-758k9\""
Nov 01 12:30:42.155034 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.154960    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 01 12:30:42.155034 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.155003    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 01 12:30:42.156303 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.156280    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"]
Nov 01 12:30:42.186681 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.186651    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krjsp\" (UniqueName: \"kubernetes.io/projected/1f347965-a09d-486c-bfa8-fa9cbe28e531-kube-api-access-krjsp\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.186813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.186685    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-maas-api-tls\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.186813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.186726    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-metrics-certs\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.186909 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.186890    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f347965-a09d-486c-bfa8-fa9cbe28e531-openshift-service-ca\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.288176 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.288148    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f347965-a09d-486c-bfa8-fa9cbe28e531-openshift-service-ca\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.288302 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.288193    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krjsp\" (UniqueName: \"kubernetes.io/projected/1f347965-a09d-486c-bfa8-fa9cbe28e531-kube-api-access-krjsp\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.288302 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.288215    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-maas-api-tls\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.288302 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.288263    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-metrics-certs\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.288829 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.288807    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f347965-a09d-486c-bfa8-fa9cbe28e531-openshift-service-ca\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.290721 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.290696    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-metrics-certs\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.290811 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.290703    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-maas-api-tls\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.342049 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.341993    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krjsp\" (UniqueName: \"kubernetes.io/projected/1f347965-a09d-486c-bfa8-fa9cbe28e531-kube-api-access-krjsp\") pod \"maas-api-6cdfff487-4z6sl\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") " pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.366813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.366783    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:42.545960 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.545928    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"]
Nov 01 12:30:42.549441 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.549427    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.600007 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.599928    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 01 12:30:42.638593 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.638567    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"]
Nov 01 12:30:42.683414 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.683382    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-bc8bt\""
Nov 01 12:30:42.692357 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.692333    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.692475 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.692364    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thxzp\" (UniqueName: \"kubernetes.io/projected/490cb0f8-5280-4505-9141-43cd22eb0cac-kube-api-access-thxzp\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.692515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.692467    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.692515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.692491    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490cb0f8-5280-4505-9141-43cd22eb0cac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.692583 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.692524    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.692617 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.692586    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.692617 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.692607    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.692700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.692687    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.692746 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.692734    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.790811 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.790790    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"]
Nov 01 12:30:42.792118 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:30:42.792087    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f347965_a09d_486c_bfa8_fa9cbe28e531.slice/crio-adac5a9d0b9f93586f59392481570ce656b1e840be6d2a799b44249c63312875 WatchSource:0}: Error finding container adac5a9d0b9f93586f59392481570ce656b1e840be6d2a799b44249c63312875: Status 404 returned error can't find the container with id adac5a9d0b9f93586f59392481570ce656b1e840be6d2a799b44249c63312875
Nov 01 12:30:42.793170 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793145    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793282 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793182    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490cb0f8-5280-4505-9141-43cd22eb0cac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793282 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793206    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793282 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793255    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793282 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793271    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793288    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793335    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793366    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793400    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thxzp\" (UniqueName: \"kubernetes.io/projected/490cb0f8-5280-4505-9141-43cd22eb0cac-kube-api-access-thxzp\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793686 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793549    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793686 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793605    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793686 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793665    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793936 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793883    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/490cb0f8-5280-4505-9141-43cd22eb0cac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.793936 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.793885    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.795843 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.795822    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.795943 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.795869    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.848799 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.848768    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/490cb0f8-5280-4505-9141-43cd22eb0cac-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.848949 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.848932    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thxzp\" (UniqueName: \"kubernetes.io/projected/490cb0f8-5280-4505-9141-43cd22eb0cac-kube-api-access-thxzp\") pod \"maas-default-gateway-openshift-default-7df854bf67-gjsds\" (UID: \"490cb0f8-5280-4505-9141-43cd22eb0cac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:42.861751 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:42.861727    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:43.271243 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:43.271217    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"]
Nov 01 12:30:43.272152 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:30:43.272128    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod490cb0f8_5280_4505_9141_43cd22eb0cac.slice/crio-5cab1ee84f7144046e1b25cb5244d034db226087b91efc6dfacb68db79ee6939 WatchSource:0}: Error finding container 5cab1ee84f7144046e1b25cb5244d034db226087b91efc6dfacb68db79ee6939: Status 404 returned error can't find the container with id 5cab1ee84f7144046e1b25cb5244d034db226087b91efc6dfacb68db79ee6939
Nov 01 12:30:43.318133 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:43.318098    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" event={"ID":"490cb0f8-5280-4505-9141-43cd22eb0cac","Type":"ContainerStarted","Data":"5cab1ee84f7144046e1b25cb5244d034db226087b91efc6dfacb68db79ee6939"}
Nov 01 12:30:43.319256 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:43.319229    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl" event={"ID":"1f347965-a09d-486c-bfa8-fa9cbe28e531","Type":"ContainerStarted","Data":"adac5a9d0b9f93586f59392481570ce656b1e840be6d2a799b44249c63312875"}
Nov 01 12:30:46.337104 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:46.337013    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl" event={"ID":"1f347965-a09d-486c-bfa8-fa9cbe28e531","Type":"ContainerStarted","Data":"d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1"}
Nov 01 12:30:46.337505 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:46.337138    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:46.478897 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:46.478843    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl" podStartSLOduration=1.238089928 podStartE2EDuration="4.478829277s" totalImagesPullingTime="3.240739349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:46.477166938 +0000 UTC m=+506.394962603" watchObservedRunningTime="2026-11-01 12:30:46.478829277 +0000 UTC m=+506.396624948"
Nov 01 12:30:47.534870 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:47.534839    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:30:47.535169 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:47.534906    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:30:47.535169 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:47.534944    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:30:48.347313 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:48.347277    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" event={"ID":"490cb0f8-5280-4505-9141-43cd22eb0cac","Type":"ContainerStarted","Data":"212f0a3ef7ed23ac5a439567f883af4bdbf4cf1a64bffe495066d64d6b8b9e5f"}
Nov 01 12:30:48.862672 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:48.862637    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:48.864172 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:48.864140    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gjsds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 12:30:48.864301 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:48.864209    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" podUID="490cb0f8-5280-4505-9141-43cd22eb0cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 12:30:49.134693 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:30:49.134599    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:30:49.862908 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:49.862873    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gjsds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 12:30:49.863326 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:49.862926    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" podUID="490cb0f8-5280-4505-9141-43cd22eb0cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 12:30:50.862989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:50.862930    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gjsds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 12:30:50.863421 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:50.863010    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" podUID="490cb0f8-5280-4505-9141-43cd22eb0cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 12:30:51.862597 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:51.862561    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gjsds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 12:30:51.862785 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:51.862613    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" podUID="490cb0f8-5280-4505-9141-43cd22eb0cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 12:30:52.347898 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:52.347825    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 12:30:52.458235 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:52.458184    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" podStartSLOduration=6.198292797 podStartE2EDuration="10.4581687s" totalImagesPullingTime="4.259875903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:30:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:30:48.522208012 +0000 UTC m=+508.440003679" watchObservedRunningTime="2026-11-01 12:30:52.4581687 +0000 UTC m=+512.375964367"
Nov 01 12:30:52.862298 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:52.862264    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gjsds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused" start-of-body=
Nov 01 12:30:52.862504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:52.862321    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" podUID="490cb0f8-5280-4505-9141-43cd22eb0cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": dial tcp 10.134.0.32:15021: connect: connection refused"
Nov 01 12:30:53.866250 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:53.866219    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:53.866647 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:53.866491    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:30:53.867229 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:30:53.867211    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds"
Nov 01 12:31:12.751864 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.751833    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"]
Nov 01 12:31:12.755369 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.755353    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.799735 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.799710    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"]
Nov 01 12:31:12.803347 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.803330    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.854091 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.854063    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"]
Nov 01 12:31:12.864237 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.864180    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.864382 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.864320    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z5kc\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-kube-api-access-7z5kc\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.864382 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.864362    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.864489 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.864466    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.864533 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.864504    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.864568 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.864561    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.864602 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.864586    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.864690 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.864676    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.864724 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.864716    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.906796 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.906768    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"]
Nov 01 12:31:12.965215 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965182    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.965363 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965222    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965363 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965329    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5n2fh\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-kube-api-access-5n2fh\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.965435 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965387    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965435 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965413    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965523 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965506    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965565 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965548    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.965606 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965576    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965666 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965651    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965721 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965692    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965778 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965717    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965778 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965748    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.965778 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965758    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965781    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.965931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965852    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.965931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965881    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.965931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965907    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.966170 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.965997    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.966170 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.966048    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.966170 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.966087    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:12.966170 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.966127    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z5kc\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-kube-api-access-7z5kc\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.966371 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.966266    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.966371 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.966280    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.967864 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.967841    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:12.968051 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:12.968035    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:13.020559 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.020488    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:13.020769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.020746    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z5kc\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-kube-api-access-7z5kc\") pod \"maas-default-gateway-openshift-default-7df854bf67-j62wc\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:13.067360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067334    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:13.067568 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067548    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.067613 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067585    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.067653 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067613    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5n2fh\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-kube-api-access-5n2fh\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.067697 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067652    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.067697 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067691    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.067794 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067709    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.067850 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067829    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.067895 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067867    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.067947 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.067906    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.068033 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.068011    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.068099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.068087    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.068154 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.068108    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.068229 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.068209    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.068395 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.068365    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.070589 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.070571    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.070840 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.070823    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.122802 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.122775    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.124237 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.124207    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5n2fh\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-kube-api-access-5n2fh\") pod \"maas-default-gateway-openshift-default-7df854bf67-49sgk\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.416114 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.416086    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:13.479854 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.479827    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"]
Nov 01 12:31:13.482755 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:31:13.482729    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44a51e92_f7f8_4b70_82ac_5439e2fc347f.slice/crio-a4b901cfdd806e1da9fe0e2197dc5da28beb735ab4b88f364d5cf3b810a3d664 WatchSource:0}: Error finding container a4b901cfdd806e1da9fe0e2197dc5da28beb735ab4b88f364d5cf3b810a3d664: Status 404 returned error can't find the container with id a4b901cfdd806e1da9fe0e2197dc5da28beb735ab4b88f364d5cf3b810a3d664
Nov 01 12:31:13.484880 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.484855    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:31:13.484964 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.484913    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:31:13.484964 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.484943    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:31:13.787412 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.787387    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"]
Nov 01 12:31:13.787753 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:31:13.787731    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed0cde7_7d50_43da_b493_bad0c76f03e7.slice/crio-fa776965b87b01c40af54c1b98c4ff21d53f3a0a3d70fb26da218515de1206b3 WatchSource:0}: Error finding container fa776965b87b01c40af54c1b98c4ff21d53f3a0a3d70fb26da218515de1206b3: Status 404 returned error can't find the container with id fa776965b87b01c40af54c1b98c4ff21d53f3a0a3d70fb26da218515de1206b3
Nov 01 12:31:13.789988 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.789949    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:31:13.790101 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.790029    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:31:13.790101 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:13.790061    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:31:14.463545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:14.463510    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" event={"ID":"44a51e92-f7f8-4b70-82ac-5439e2fc347f","Type":"ContainerStarted","Data":"17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085"}
Nov 01 12:31:14.463733 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:14.463558    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" event={"ID":"44a51e92-f7f8-4b70-82ac-5439e2fc347f","Type":"ContainerStarted","Data":"a4b901cfdd806e1da9fe0e2197dc5da28beb735ab4b88f364d5cf3b810a3d664"}
Nov 01 12:31:14.465043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:14.465020    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" event={"ID":"0ed0cde7-7d50-43da-b493-bad0c76f03e7","Type":"ContainerStarted","Data":"5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7"}
Nov 01 12:31:14.465146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:14.465059    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" event={"ID":"0ed0cde7-7d50-43da-b493-bad0c76f03e7","Type":"ContainerStarted","Data":"fa776965b87b01c40af54c1b98c4ff21d53f3a0a3d70fb26da218515de1206b3"}
Nov 01 12:31:14.698524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:14.698473    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" podStartSLOduration=2.698457771 podStartE2EDuration="2.698457771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:31:14.697615525 +0000 UTC m=+534.615411193" watchObservedRunningTime="2026-11-01 12:31:14.698457771 +0000 UTC m=+534.616253436"
Nov 01 12:31:14.699099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:14.699073    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" podStartSLOduration=2.6990635689999998 podStartE2EDuration="2.699063569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:31:14.584953542 +0000 UTC m=+534.502749210" watchObservedRunningTime="2026-11-01 12:31:14.699063569 +0000 UTC m=+534.616859236"
Nov 01 12:31:15.067861 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:15.067822    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:15.068962 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:15.068937    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j62wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:31:15.069105 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:15.069006    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 01 12:31:15.419312 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:15.419280    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:15.430146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:15.430110    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-49sgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 12:31:15.430423 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:15.430393    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 12:31:16.068268 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:16.068239    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j62wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:31:16.068728 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:16.068702    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 01 12:31:16.418860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:16.418407    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-49sgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 12:31:16.418860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:16.418472    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 12:31:17.068333 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:17.068284    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j62wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:31:17.068733 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:17.068360    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 01 12:31:17.419995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:17.419421    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-49sgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 12:31:17.419995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:17.419483    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 12:31:18.067741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:18.067709    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j62wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:31:18.067948 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:18.067760    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 01 12:31:18.419995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:18.419379    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-49sgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 12:31:18.419995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:18.419452    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 12:31:19.068064 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:19.068032    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j62wc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused" start-of-body=
Nov 01 12:31:19.068246 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:19.068086    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": dial tcp 10.134.0.33:15021: connect: connection refused"
Nov 01 12:31:19.156015 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:31:19.155135    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:31:19.418513 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:19.418474    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-49sgk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused" start-of-body=
Nov 01 12:31:19.418937 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:19.418905    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": dial tcp 10.134.0.34:15021: connect: connection refused"
Nov 01 12:31:20.072031 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:20.071999    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:20.072466 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:20.072390    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:20.073173 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:20.073154    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 12:31:20.420519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:20.420490    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:20.420835 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:20.420815    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:20.421457 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:31:20.421440    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:31:49.111844 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:31:49.111804    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:32:19.112481 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:32:19.112440    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:32:20.220729 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:20.220698    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:32:20.222763 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:20.222736    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:32:49.111781 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:32:49.111743    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:32:49.710587 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.710557    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"]
Nov 01 12:32:49.714166 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.714148    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.764291 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.764255    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 01 12:32:49.809688 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.809652    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.809860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.809694    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.809860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.809728    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.809860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.809846    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.809982 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.809874    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzgp\" (UniqueName: \"kubernetes.io/projected/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-kube-api-access-kzzgp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.810020 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.809966    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.811377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.811357    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-5s8np\""
Nov 01 12:32:49.811461 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.811358    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 01 12:32:49.811461 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.811394    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 01 12:32:49.813291 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.813267    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"]
Nov 01 12:32:49.910813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.910783    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.910813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.910820    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.911081 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.910838    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.911081 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.910922    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.911081 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.910954    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzgp\" (UniqueName: \"kubernetes.io/projected/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-kube-api-access-kzzgp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.911081 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.911074    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.911313 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.911291    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.911391 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.911368    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.911496 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.911477    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.913121 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.913097    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.913415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.913400    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:49.968744 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:49.968672    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzgp\" (UniqueName: \"kubernetes.io/projected/6bee1b64-413a-4f04-b05a-b7ad5acda1a2-kube-api-access-kzzgp\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7\" (UID: \"6bee1b64-413a-4f04-b05a-b7ad5acda1a2\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:50.026137 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:50.026104    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:50.395051 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:50.395028    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"]
Nov 01 12:32:50.397317 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:32:50.397288    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bee1b64_413a_4f04_b05a_b7ad5acda1a2.slice/crio-8870ad5cb39508a10dfa0a7122f6a21bd54a85d48d4a41341519bde58abafd52 WatchSource:0}: Error finding container 8870ad5cb39508a10dfa0a7122f6a21bd54a85d48d4a41341519bde58abafd52: Status 404 returned error can't find the container with id 8870ad5cb39508a10dfa0a7122f6a21bd54a85d48d4a41341519bde58abafd52
Nov 01 12:32:50.900934 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:50.900899    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7" event={"ID":"6bee1b64-413a-4f04-b05a-b7ad5acda1a2","Type":"ContainerStarted","Data":"8870ad5cb39508a10dfa0a7122f6a21bd54a85d48d4a41341519bde58abafd52"}
Nov 01 12:32:52.917457 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:52.917420    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7" event={"ID":"6bee1b64-413a-4f04-b05a-b7ad5acda1a2","Type":"ContainerStarted","Data":"0da6b365a44934530be2c728d826d7b69b7c919518aeaf212bab71e748252933"}
Nov 01 12:32:53.025898 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:53.025844    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7" podStartSLOduration=2.422431254 podStartE2EDuration="4.025830619s" totalImagesPullingTime="1.603399365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:32:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:32:53.025640058 +0000 UTC m=+632.943435723" watchObservedRunningTime="2026-11-01 12:32:53.025830619 +0000 UTC m=+632.943626284"
Nov 01 12:32:53.026641 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:53.026618    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:32:57.862618 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.862582    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"]
Nov 01 12:32:57.866513 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.866498    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:57.921010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.920847    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Nov 01 12:32:57.966314 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.966288    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"]
Nov 01 12:32:57.994410 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.994383    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:57.994562 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.994418    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb9gb\" (UniqueName: \"kubernetes.io/projected/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-kube-api-access-bb9gb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:57.994562 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.994452    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:57.994641 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.994553    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:57.994641 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.994598    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:57.994733 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:57.994712    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.095701 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.095666    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.095877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.095714    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.095877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.095757    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.095877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.095788    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.095877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.095807    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb9gb\" (UniqueName: \"kubernetes.io/projected/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-kube-api-access-bb9gb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.095877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.095843    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.096212 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.096191    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.096283 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.096215    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.096283 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.096269    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.098113 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.098097    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.098365 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.098349    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.149629 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.149563    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb9gb\" (UniqueName: \"kubernetes.io/projected/e6bf57e8-4c6b-4a61-ad73-650f4d6a8696-kube-api-access-bb9gb\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-4r49f\" (UID: \"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.179486 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.179454    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:32:58.561190 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.561167    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"]
Nov 01 12:32:58.562591 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:32:58.562551    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6bf57e8_4c6b_4a61_ad73_650f4d6a8696.slice/crio-a68e73a6ce4ec1818ad35cdcf8772d8da15fcf72ce6fcf3ceb760065f7375d54 WatchSource:0}: Error finding container a68e73a6ce4ec1818ad35cdcf8772d8da15fcf72ce6fcf3ceb760065f7375d54: Status 404 returned error can't find the container with id a68e73a6ce4ec1818ad35cdcf8772d8da15fcf72ce6fcf3ceb760065f7375d54
Nov 01 12:32:58.951583 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:58.951550    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f" event={"ID":"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696","Type":"ContainerStarted","Data":"a68e73a6ce4ec1818ad35cdcf8772d8da15fcf72ce6fcf3ceb760065f7375d54"}
Nov 01 12:32:59.957147 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:32:59.957112    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f" event={"ID":"e6bf57e8-4c6b-4a61-ad73-650f4d6a8696","Type":"ContainerStarted","Data":"9c738ae4219f0188d4b144169d69bce67f5069587bf1d463cb78077df6444066"}
Nov 01 12:33:00.026599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:00.026570    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:33:00.039082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:00.039058    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:33:00.066844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:00.066797    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f" podStartSLOduration=2.468905922 podStartE2EDuration="3.066782885s" totalImagesPullingTime="597.876963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:32:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:33:00.065442127 +0000 UTC m=+639.983237793" watchObservedRunningTime="2026-11-01 12:33:00.066782885 +0000 UTC m=+639.984578584"
Nov 01 12:33:00.180933 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:00.180843    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:33:00.973463 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:00.973434    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7"
Nov 01 12:33:06.450219 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.450189    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"]
Nov 01 12:33:06.453668 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.453649    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.494887 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.494857    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.495043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.494897    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.495043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.495019    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.495200 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.495070    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ljvz\" (UniqueName: \"kubernetes.io/projected/68b51645-28d7-42ff-8aa1-0e4d49c02444-kube-api-access-4ljvz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.495200 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.495144    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b51645-28d7-42ff-8aa1-0e4d49c02444-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.495200 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.495188    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.508955 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.508930    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"]
Nov 01 12:33:06.553763 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.553736    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Nov 01 12:33:06.596610 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.596576    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.596610 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.596615    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.596872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.596657    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.596872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.596688    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ljvz\" (UniqueName: \"kubernetes.io/projected/68b51645-28d7-42ff-8aa1-0e4d49c02444-kube-api-access-4ljvz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.596872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.596720    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b51645-28d7-42ff-8aa1-0e4d49c02444-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.596872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.596760    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.597146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.597123    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.597197 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.597163    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.597273 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.597255    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.599277 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.599251    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68b51645-28d7-42ff-8aa1-0e4d49c02444-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.599806 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.599777    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68b51645-28d7-42ff-8aa1-0e4d49c02444-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.650661 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.650621    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ljvz\" (UniqueName: \"kubernetes.io/projected/68b51645-28d7-42ff-8aa1-0e4d49c02444-kube-api-access-4ljvz\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh\" (UID: \"68b51645-28d7-42ff-8aa1-0e4d49c02444\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:06.766399 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:06.766313    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:07.132460 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:07.132435    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"]
Nov 01 12:33:07.136469 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:33:07.136437    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b51645_28d7_42ff_8aa1_0e4d49c02444.slice/crio-0e319771ff7f97d2df74e0797f8e88a9bc5b28df4b4fd650df6e089c38c15548 WatchSource:0}: Error finding container 0e319771ff7f97d2df74e0797f8e88a9bc5b28df4b4fd650df6e089c38c15548: Status 404 returned error can't find the container with id 0e319771ff7f97d2df74e0797f8e88a9bc5b28df4b4fd650df6e089c38c15548
Nov 01 12:33:08.000235 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:08.000144    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh" event={"ID":"68b51645-28d7-42ff-8aa1-0e4d49c02444","Type":"ContainerStarted","Data":"e0b8660f28044de6ce13dd3881df8df206240ddb7e6ed1650426cb00ac715474"}
Nov 01 12:33:08.000235 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:08.000194    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh" event={"ID":"68b51645-28d7-42ff-8aa1-0e4d49c02444","Type":"ContainerStarted","Data":"0e319771ff7f97d2df74e0797f8e88a9bc5b28df4b4fd650df6e089c38c15548"}
Nov 01 12:33:08.111159 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:08.111106    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh" podStartSLOduration=1.500893493 podStartE2EDuration="2.111090059s" totalImagesPullingTime="610.196566ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:33:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:33:08.108743443 +0000 UTC m=+648.026539110" watchObservedRunningTime="2026-11-01 12:33:08.111090059 +0000 UTC m=+648.028885753"
Nov 01 12:33:08.180142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:08.180113    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:33:08.192672 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:08.192650    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:33:08.767162 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:08.767127    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:09.033819 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:09.033739    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-4r49f"
Nov 01 12:33:13.192203 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.192164    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"]
Nov 01 12:33:13.196048 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.196026    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.278300 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.278271    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"]
Nov 01 12:33:13.294912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.294887    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.295022 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.294926    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.295063 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.295022    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd2mh\" (UniqueName: \"kubernetes.io/projected/9086be84-9c9e-4501-82ca-a580e08b3fd0-kube-api-access-jd2mh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.295158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.295139    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.295193 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.295170    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9086be84-9c9e-4501-82ca-a580e08b3fd0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.295252 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.295235    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.295387 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.295372    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 01 12:33:13.396782 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.396748    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.396782 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.396785    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9086be84-9c9e-4501-82ca-a580e08b3fd0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.397063 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.396804    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.397063 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.396873    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.397063 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.396901    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.397063 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.396941    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd2mh\" (UniqueName: \"kubernetes.io/projected/9086be84-9c9e-4501-82ca-a580e08b3fd0-kube-api-access-jd2mh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.397309 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.397288    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.397362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.397313    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.397435 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.397404    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.399071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.399048    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9086be84-9c9e-4501-82ca-a580e08b3fd0-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.399342 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.399317    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9086be84-9c9e-4501-82ca-a580e08b3fd0-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.451089 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.451007    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd2mh\" (UniqueName: \"kubernetes.io/projected/9086be84-9c9e-4501-82ca-a580e08b3fd0-kube-api-access-jd2mh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf\" (UID: \"9086be84-9c9e-4501-82ca-a580e08b3fd0\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.509155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.509113    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:13.870882 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:13.870856    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"]
Nov 01 12:33:13.871674 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:33:13.871643    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9086be84_9c9e_4501_82ca_a580e08b3fd0.slice/crio-8080062c8ada34942a394c236525731b4cfc1bf2cd2e61abc397e0dae7a0c23e WatchSource:0}: Error finding container 8080062c8ada34942a394c236525731b4cfc1bf2cd2e61abc397e0dae7a0c23e: Status 404 returned error can't find the container with id 8080062c8ada34942a394c236525731b4cfc1bf2cd2e61abc397e0dae7a0c23e
Nov 01 12:33:14.086936 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:14.086901    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf" event={"ID":"9086be84-9c9e-4501-82ca-a580e08b3fd0","Type":"ContainerStarted","Data":"8080062c8ada34942a394c236525731b4cfc1bf2cd2e61abc397e0dae7a0c23e"}
Nov 01 12:33:15.093078 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:15.093033    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf" event={"ID":"9086be84-9c9e-4501-82ca-a580e08b3fd0","Type":"ContainerStarted","Data":"092b437884da112d563e642a7cc31bf2ad3b5a51e2324a096a282bfb871ff113"}
Nov 01 12:33:15.202856 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:15.202807    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf" podStartSLOduration=1.5981125010000001 podStartE2EDuration="2.202789826s" totalImagesPullingTime="604.677325ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:33:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:33:15.201649482 +0000 UTC m=+655.119445149" watchObservedRunningTime="2026-11-01 12:33:15.202789826 +0000 UTC m=+655.120585494"
Nov 01 12:33:15.509923 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:15.509876    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:16.766769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:16.766729    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:16.786308 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:16.786276    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:17.115934 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:17.115906    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh"
Nov 01 12:33:19.113403 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:33:19.113363    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:33:23.509938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:23.509909    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:23.522431 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:23.522405    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:24.163179 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:33:24.163153    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf"
Nov 01 12:33:49.114243 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:33:49.114167    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:34:19.113452 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:34:19.113414    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:34:42.623753 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.623718    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"]
Nov 01 12:34:42.627479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.627460    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.657393 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.657366    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"]
Nov 01 12:34:42.680201 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.680174    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-94c94b-openshift-default-dockercfg-r5knk\""
Nov 01 12:34:42.753413 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.753381    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.753572 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.753429    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.753572 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.753494    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.753572 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.753516    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.753572 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.753540    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whhh6\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-kube-api-access-whhh6\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.753707 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.753639    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.753707 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.753667    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.753707 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.753688    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.753824 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.753746    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855240 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855208    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855447 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855320    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855447 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855377    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855447 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855412    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855612 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855444    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855612 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855486    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whhh6\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-kube-api-access-whhh6\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855612 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855528    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855612 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855556    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855817 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855654    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855817 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855707    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.855817 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855724    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.856016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.855933    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.856065 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.856041    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.856148 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.856127    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.857810 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.857786    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.857940 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.857921    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.912916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.912854    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.913147 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.913127    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whhh6\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-kube-api-access-whhh6\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:42.941539 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:42.941498    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:43.218405 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.218337    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"]
Nov 01 12:34:43.223117 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.223097    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.272936 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.272898    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-06fd6b-openshift-default-dockercfg-6x29r\""
Nov 01 12:34:43.280196 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.280164    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"]
Nov 01 12:34:43.362055 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.362025    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.362283 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.362095    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d15666-ddd7-4270-8e31-4630a4b89b39-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.362283 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.362233    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.362415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.362281    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.362415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.362396    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.362415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.362416    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.362594 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.362527    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.362594 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.362552    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tx46\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-kube-api-access-2tx46\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.362675 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.362642    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.368808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.368785    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"]
Nov 01 12:34:43.381838 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:34:43.381815    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc7262d_f0eb_49c8_8a2f_f859df7de7f2.slice/crio-80f3118854a29bc974fd38df0958f310d5e0f5d7a0ff73384fb04dd7cd928cb1 WatchSource:0}: Error finding container 80f3118854a29bc974fd38df0958f310d5e0f5d7a0ff73384fb04dd7cd928cb1: Status 404 returned error can't find the container with id 80f3118854a29bc974fd38df0958f310d5e0f5d7a0ff73384fb04dd7cd928cb1
Nov 01 12:34:43.383916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.383884    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:34:43.384014 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.383965    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:34:43.384081 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.384066    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:34:43.463659 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.463622    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.463837 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.463684    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tx46\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-kube-api-access-2tx46\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.463837 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.463742    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.463944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.463869    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.463944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.463912    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d15666-ddd7-4270-8e31-4630a4b89b39-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.464079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.463947    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.464079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.464004    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.464079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.464066    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.464227 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.464093    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.464227 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.464122    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.464227 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.464145    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.464470 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.464450    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.464671 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.464650    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d15666-ddd7-4270-8e31-4630a4b89b39-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.464797 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.464774    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.466518 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.466422    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.466806 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.466789    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.521253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.521176    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tx46\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-kube-api-access-2tx46\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.521832 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.521680    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.522435 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.522403    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" event={"ID":"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2","Type":"ContainerStarted","Data":"e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e"}
Nov 01 12:34:43.522558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.522461    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" event={"ID":"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2","Type":"ContainerStarted","Data":"80f3118854a29bc974fd38df0958f310d5e0f5d7a0ff73384fb04dd7cd928cb1"}
Nov 01 12:34:43.536006 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.535983    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:43.643785 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.643725    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" podStartSLOduration=1.643703197 podStartE2EDuration="1.643703197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:43.642608273 +0000 UTC m=+743.560403943" watchObservedRunningTime="2026-11-01 12:34:43.643703197 +0000 UTC m=+743.561498865"
Nov 01 12:34:43.899436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.899405    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"]
Nov 01 12:34:43.904306 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.904288    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:43.913578 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.913548    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"]
Nov 01 12:34:43.941843 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.941818    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:43.947769 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:34:43.947728    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94d15666_ddd7_4270_8e31_4630a4b89b39.slice/crio-296120bbaad9c875ad497d05fab15555e8efa1c3f32060b5f1465e19d6b7af7e WatchSource:0}: Error finding container 296120bbaad9c875ad497d05fab15555e8efa1c3f32060b5f1465e19d6b7af7e: Status 404 returned error can't find the container with id 296120bbaad9c875ad497d05fab15555e8efa1c3f32060b5f1465e19d6b7af7e
Nov 01 12:34:43.950611 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.950577    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:34:43.950710 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.950668    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:34:43.950759 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.950714    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:34:43.954684 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.954662    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-5b1bd4-openshift-default-dockercfg-4g7mc\""
Nov 01 12:34:43.960266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:43.960245    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"]
Nov 01 12:34:44.071127 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.071090    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.071288 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.071151    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09815128-1266-4fed-b6a1-d059cc352fc4-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.071330 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.071287    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.071422 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.071405    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.071458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.071439    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.071494 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.071480    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.071532 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.071519    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09815128-1266-4fed-b6a1-d059cc352fc4-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.071622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.071605    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.071661 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.071636    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbhhl\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-kube-api-access-dbhhl\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172397 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172304    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172397 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172367    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172397 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172392    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172677 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172410    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172677 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172434    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09815128-1266-4fed-b6a1-d059cc352fc4-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172677 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172473    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172677 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172580    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbhhl\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-kube-api-access-dbhhl\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172677 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172639    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172691    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09815128-1266-4fed-b6a1-d059cc352fc4-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.172924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.172802    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.173061 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.173016    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.173114 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.173065    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.173371 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.173349    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.173504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.173481    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09815128-1266-4fed-b6a1-d059cc352fc4-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.175062 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.175032    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.175416 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.175395    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09815128-1266-4fed-b6a1-d059cc352fc4-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.233925 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.233894    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.269635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.269566    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbhhl\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-kube-api-access-dbhhl\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.519202 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.519111    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:44.530508 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.530475    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" event={"ID":"94d15666-ddd7-4270-8e31-4630a4b89b39","Type":"ContainerStarted","Data":"22236c497292401b9ca72a58439df4ddc2238762a796794ded066bc01690895c"}
Nov 01 12:34:44.530664 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.530528    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" event={"ID":"94d15666-ddd7-4270-8e31-4630a4b89b39","Type":"ContainerStarted","Data":"296120bbaad9c875ad497d05fab15555e8efa1c3f32060b5f1465e19d6b7af7e"}
Nov 01 12:34:44.536918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.536897    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:44.663988 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.663925    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" podStartSLOduration=1.663910541 podStartE2EDuration="1.663910541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:44.662124299 +0000 UTC m=+744.579919979" watchObservedRunningTime="2026-11-01 12:34:44.663910541 +0000 UTC m=+744.581706206"
Nov 01 12:34:44.930648 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.930619    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"]
Nov 01 12:34:44.932229 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:34:44.932203    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09815128_1266_4fed_b6a1_d059cc352fc4.slice/crio-a7f285f4e1872cfb660276dd5184b0cfe343b1d54227f007013b5d95dba2119a WatchSource:0}: Error finding container a7f285f4e1872cfb660276dd5184b0cfe343b1d54227f007013b5d95dba2119a: Status 404 returned error can't find the container with id a7f285f4e1872cfb660276dd5184b0cfe343b1d54227f007013b5d95dba2119a
Nov 01 12:34:44.934789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.934757    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:34:44.934863 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.934837    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:34:44.934901 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.934884    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:34:44.947531 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:44.947502    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:45.536657 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:45.536623    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" event={"ID":"09815128-1266-4fed-b6a1-d059cc352fc4","Type":"ContainerStarted","Data":"198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756"}
Nov 01 12:34:45.536822 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:45.536671    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" event={"ID":"09815128-1266-4fed-b6a1-d059cc352fc4","Type":"ContainerStarted","Data":"a7f285f4e1872cfb660276dd5184b0cfe343b1d54227f007013b5d95dba2119a"}
Nov 01 12:34:45.537101 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:45.537061    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:45.538053 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:45.538035    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:34:45.540914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:45.540896    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:45.650031 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:45.649983    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" podStartSLOduration=2.649956074 podStartE2EDuration="2.649956074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:45.648248516 +0000 UTC m=+745.566044183" watchObservedRunningTime="2026-11-01 12:34:45.649956074 +0000 UTC m=+745.567751740"
Nov 01 12:34:46.519451 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:46.519421    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:46.524429 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:46.524394    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:46.542025 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:46.542000    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:46.542142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:46.542048    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:46.542472 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:46.542455    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:34:46.542552 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:46.542499    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:34:49.114068 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:34:49.114034    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:34:53.075774 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.075741    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"]
Nov 01 12:34:53.080490 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.080471    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.132524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.132494    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-94c94b-serving-cert\""
Nov 01 12:34:53.184776 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.184743    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"]
Nov 01 12:34:53.267498 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.267463    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2915-6553-4a33-8291-4d930d27104c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.267672 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.267617    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpj68\" (UniqueName: \"kubernetes.io/projected/94fd2915-6553-4a33-8291-4d930d27104c-kube-api-access-rpj68\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.267809 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.267698    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.267864 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.267829    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-metrics-certs\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.369062 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.369030    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2915-6553-4a33-8291-4d930d27104c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.369252 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.369151    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpj68\" (UniqueName: \"kubernetes.io/projected/94fd2915-6553-4a33-8291-4d930d27104c-kube-api-access-rpj68\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.369333 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.369288    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.369402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.369346    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-metrics-certs\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.369769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.369747    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2915-6553-4a33-8291-4d930d27104c-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.371579 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.371560    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-metrics-certs\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.371721 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.371700    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-maas-api-tls\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.423804 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.423775    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpj68\" (UniqueName: \"kubernetes.io/projected/94fd2915-6553-4a33-8291-4d930d27104c-kube-api-access-rpj68\") pod \"maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:53.693424 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:53.693333    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:54.059178 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:34:54.059142    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fd2915_6553_4a33_8291_4d930d27104c.slice/crio-defdc6eccd23395222088c3032cfa42bc9f2e7f7bc9aad709c4c396694f2f524 WatchSource:0}: Error finding container defdc6eccd23395222088c3032cfa42bc9f2e7f7bc9aad709c4c396694f2f524: Status 404 returned error can't find the container with id defdc6eccd23395222088c3032cfa42bc9f2e7f7bc9aad709c4c396694f2f524
Nov 01 12:34:54.060609 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:54.060591    4095 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:34:54.086622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:54.086591    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"]
Nov 01 12:34:54.580564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:54.580529    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf" event={"ID":"94fd2915-6553-4a33-8291-4d930d27104c","Type":"ContainerStarted","Data":"defdc6eccd23395222088c3032cfa42bc9f2e7f7bc9aad709c4c396694f2f524"}
Nov 01 12:34:55.587174 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:55.587139    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf" event={"ID":"94fd2915-6553-4a33-8291-4d930d27104c","Type":"ContainerStarted","Data":"aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9"}
Nov 01 12:34:55.587579 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:55.587217    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:34:55.697087 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:55.697037    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf" podStartSLOduration=1.7742181970000002 podStartE2EDuration="2.69702445s" totalImagesPullingTime="922.806253ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:34:55.695649038 +0000 UTC m=+755.613444705" watchObservedRunningTime="2026-11-01 12:34:55.69702445 +0000 UTC m=+755.614820115"
Nov 01 12:34:56.289877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.289844    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"]
Nov 01 12:34:56.293649 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.293633    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:34:56.297914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.297887    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616d14ef-ae4d-40b4-8103-2a42e3613f74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt\" (UID: \"616d14ef-ae4d-40b4-8103-2a42e3613f74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:34:56.298086 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.297922    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cchgk\" (UniqueName: \"kubernetes.io/projected/616d14ef-ae4d-40b4-8103-2a42e3613f74-kube-api-access-cchgk\") pod \"payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt\" (UID: \"616d14ef-ae4d-40b4-8103-2a42e3613f74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:34:56.344738 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.344702    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-94c94b\""
Nov 01 12:34:56.345037 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.345012    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-94c94b-dockercfg-8244r\""
Nov 01 12:34:56.347029 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.347001    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"]
Nov 01 12:34:56.399234 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.399197    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616d14ef-ae4d-40b4-8103-2a42e3613f74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt\" (UID: \"616d14ef-ae4d-40b4-8103-2a42e3613f74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:34:56.400128 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.400102    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cchgk\" (UniqueName: \"kubernetes.io/projected/616d14ef-ae4d-40b4-8103-2a42e3613f74-kube-api-access-cchgk\") pod \"payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt\" (UID: \"616d14ef-ae4d-40b4-8103-2a42e3613f74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:34:56.400581 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.400534    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616d14ef-ae4d-40b4-8103-2a42e3613f74-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt\" (UID: \"616d14ef-ae4d-40b4-8103-2a42e3613f74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:34:56.455548 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.455515    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cchgk\" (UniqueName: \"kubernetes.io/projected/616d14ef-ae4d-40b4-8103-2a42e3613f74-kube-api-access-cchgk\") pod \"payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt\" (UID: \"616d14ef-ae4d-40b4-8103-2a42e3613f74\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:34:56.607089 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.607062    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:34:56.969720 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:56.969697    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"]
Nov 01 12:34:56.981473 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:34:56.971944    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod616d14ef_ae4d_40b4_8103_2a42e3613f74.slice/crio-7e97c4338208abeeac787dde2c3d5a6776ee615954ed78453c536cb863309d40 WatchSource:0}: Error finding container 7e97c4338208abeeac787dde2c3d5a6776ee615954ed78453c536cb863309d40: Status 404 returned error can't find the container with id 7e97c4338208abeeac787dde2c3d5a6776ee615954ed78453c536cb863309d40
Nov 01 12:34:57.598329 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:57.598296    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt" event={"ID":"616d14ef-ae4d-40b4-8103-2a42e3613f74","Type":"ContainerStarted","Data":"7e97c4338208abeeac787dde2c3d5a6776ee615954ed78453c536cb863309d40"}
Nov 01 12:34:58.484783 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.480097    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"]
Nov 01 12:34:58.484783 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.484658    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.524525 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.524493    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj9bf\" (UniqueName: \"kubernetes.io/projected/9d05dbc0-db8f-4084-874e-93b23b734511-kube-api-access-kj9bf\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.524725 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.524534    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d05dbc0-db8f-4084-874e-93b23b734511-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.524725 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.524656    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.524725 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.524712    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-metrics-certs\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.534542 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.534514    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-c7206f-serving-cert\""
Nov 01 12:34:58.585629 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.585592    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"]
Nov 01 12:34:58.627135 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.626147    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.627135 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.626227    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-metrics-certs\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.627135 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:34:58.626304    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-serving-cert: secret "maas-api-e2e-worker-w2-c7206f-serving-cert" not found
Nov 01 12:34:58.627135 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.626328    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj9bf\" (UniqueName: \"kubernetes.io/projected/9d05dbc0-db8f-4084-874e-93b23b734511-kube-api-access-kj9bf\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.627135 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.626358    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d05dbc0-db8f-4084-874e-93b23b734511-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.627135 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:34:58.626376    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls podName:9d05dbc0-db8f-4084-874e-93b23b734511 nodeName:}" failed. No retries permitted until 2026-11-01 12:34:59.126358326 +0000 UTC m=+759.044154005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls") pod "maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs" (UID: "9d05dbc0-db8f-4084-874e-93b23b734511") : secret "maas-api-e2e-worker-w2-c7206f-serving-cert" not found
Nov 01 12:34:58.627135 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.627086    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d05dbc0-db8f-4084-874e-93b23b734511-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.630302 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.630271    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-metrics-certs\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:58.683894 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:58.683861    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj9bf\" (UniqueName: \"kubernetes.io/projected/9d05dbc0-db8f-4084-874e-93b23b734511-kube-api-access-kj9bf\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:59.131408 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:59.131373    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:59.134053 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:59.134023    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls\") pod \"maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:59.403304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:59.403230    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:34:59.779199 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:34:59.779167    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"]
Nov 01 12:34:59.779965 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:34:59.779934    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d05dbc0_db8f_4084_874e_93b23b734511.slice/crio-53c391fb0613fd698d1313dcf7f70a16ee15b653bc40389f33870119d20afd66 WatchSource:0}: Error finding container 53c391fb0613fd698d1313dcf7f70a16ee15b653bc40389f33870119d20afd66: Status 404 returned error can't find the container with id 53c391fb0613fd698d1313dcf7f70a16ee15b653bc40389f33870119d20afd66
Nov 01 12:35:00.620966 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:00.620937    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt" event={"ID":"616d14ef-ae4d-40b4-8103-2a42e3613f74","Type":"ContainerStarted","Data":"edaf5cf12d8e86073d5302f2b17ad8b42ea66e1d166f3f69f14e77a4c29897a4"}
Nov 01 12:35:00.621152 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:00.621061    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:35:00.622279 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:00.622254    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs" event={"ID":"9d05dbc0-db8f-4084-874e-93b23b734511","Type":"ContainerStarted","Data":"53c391fb0613fd698d1313dcf7f70a16ee15b653bc40389f33870119d20afd66"}
Nov 01 12:35:00.729349 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:00.729302    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt" podStartSLOduration=1.7749720020000002 podStartE2EDuration="4.729288551s" totalImagesPullingTime="2.954316549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:00.727919805 +0000 UTC m=+760.645715470" watchObservedRunningTime="2026-11-01 12:35:00.729288551 +0000 UTC m=+760.647084216"
Nov 01 12:35:01.598362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:01.598334    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 12:35:01.629608 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:01.629570    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs" event={"ID":"9d05dbc0-db8f-4084-874e-93b23b734511","Type":"ContainerStarted","Data":"9c1ae56ff75aa531d58dc64486ae3ebcb893a05f680fb6669eddcd7c1e153a96"}
Nov 01 12:35:01.629781 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:01.629634    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:35:01.818695 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:01.818640    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs" podStartSLOduration=2.9400370689999997 podStartE2EDuration="3.818625366s" totalImagesPullingTime="878.588297ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:34:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:01.818154892 +0000 UTC m=+761.735950559" watchObservedRunningTime="2026-11-01 12:35:01.818625366 +0000 UTC m=+761.736421031"
Nov 01 12:35:03.882752 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:03.882719    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"]
Nov 01 12:35:03.886542 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:03.886513    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:03.981674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:03.981643    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-metrics-certs\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:03.981837 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:03.981772    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6x8z\" (UniqueName: \"kubernetes.io/projected/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-kube-api-access-c6x8z\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:03.981837 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:03.981814    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:03.981914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:03.981855    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:03.984812 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:03.984794    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-5b1bd4-serving-cert\""
Nov 01 12:35:03.986943 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:03.986925    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"]
Nov 01 12:35:04.082587 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.082557    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-metrics-certs\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:04.082741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.082632    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6x8z\" (UniqueName: \"kubernetes.io/projected/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-kube-api-access-c6x8z\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:04.082741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.082665    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:04.082741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.082699    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:04.082884 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:35:04.082782    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-serving-cert: secret "maas-api-e2e-worker-w1-5b1bd4-serving-cert" not found
Nov 01 12:35:04.082884 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:35:04.082847    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls podName:6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41 nodeName:}" failed. No retries permitted until 2026-11-01 12:35:04.582826508 +0000 UTC m=+764.500622158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls") pod "maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82" (UID: "6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41") : secret "maas-api-e2e-worker-w1-5b1bd4-serving-cert" not found
Nov 01 12:35:04.083304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.083281    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:04.084948 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.084927    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-metrics-certs\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:04.136906 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.136836    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6x8z\" (UniqueName: \"kubernetes.io/projected/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-kube-api-access-c6x8z\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:04.588100 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.588016    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:04.590439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.590419    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls\") pod \"maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:04.799272 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:04.799236    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:05.189280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:05.189254    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"]
Nov 01 12:35:05.190430 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:35:05.190404    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c1a8c3e_3a2c_4050_b7c9_a0ea90db6e41.slice/crio-10cc66b5d4f37c072446af776318135a2ddf0ba6cf702780311a3686c7da063f WatchSource:0}: Error finding container 10cc66b5d4f37c072446af776318135a2ddf0ba6cf702780311a3686c7da063f: Status 404 returned error can't find the container with id 10cc66b5d4f37c072446af776318135a2ddf0ba6cf702780311a3686c7da063f
Nov 01 12:35:05.650020 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:05.649964    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82" event={"ID":"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41","Type":"ContainerStarted","Data":"10cc66b5d4f37c072446af776318135a2ddf0ba6cf702780311a3686c7da063f"}
Nov 01 12:35:06.657217 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:06.657179    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82" event={"ID":"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41","Type":"ContainerStarted","Data":"073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231"}
Nov 01 12:35:06.657684 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:06.657276    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:06.779362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:06.779311    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82" podStartSLOduration=2.912328644 podStartE2EDuration="3.779297695s" totalImagesPullingTime="866.969051ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:06.778549588 +0000 UTC m=+766.696345264" watchObservedRunningTime="2026-11-01 12:35:06.779297695 +0000 UTC m=+766.697093402"
Nov 01 12:35:07.641689 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:07.641664    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:35:11.633381 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:11.633353    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:35:11.687052 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:11.687027    4095 generic.go:356] "Generic (PLEG): container finished" podID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerID="e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e" exitCode=0
Nov 01 12:35:11.687224 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:11.687100    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" event={"ID":"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2","Type":"ContainerDied","Data":"e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e"}
Nov 01 12:35:11.687503 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:11.687489    4095 scope.go:122] "RemoveContainer" containerID="e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e"
Nov 01 12:35:11.688866 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:11.688844    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:11.688927 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:11.688909    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:11.688962 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:11.688942    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:12.668174 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:12.668142    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 12:35:12.696897 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:12.696854    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" event={"ID":"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2","Type":"ContainerStarted","Data":"753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2"}
Nov 01 12:35:12.942587 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:12.942499    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:35:12.943936 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:12.943908    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 12:35:12.944071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:12.943962    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 12:35:13.253502 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.253424    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"]
Nov 01 12:35:13.256522 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.256506    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.341194 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.341144    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"]
Nov 01 12:35:13.397466 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.397431    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-c7206f-openshift-default-dockercfg-8pj7g\""
Nov 01 12:35:13.457953 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.457921    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.458134 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.457966    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17bdbf6-1545-43fd-8053-92a61f75edc1-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.458134 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.458080    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.458134 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.458125    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.458287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.458150    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btr6j\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-kube-api-access-btr6j\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.458287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.458207    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.458366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.458287    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.458366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.458314    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.458366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.458330    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.559393 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.559303    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.559393 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.559379    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.559701 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.559412    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btr6j\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-kube-api-access-btr6j\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.559701 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.559435    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.559701 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.559500    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.559701 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.559535    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.559701 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.559558    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.559701 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.559634    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.561393 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.559726    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17bdbf6-1545-43fd-8053-92a61f75edc1-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.561393 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.561013    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17bdbf6-1545-43fd-8053-92a61f75edc1-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.561393 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.561320    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.561736 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.561713    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.562621 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.562594    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.563938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.563910    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.569513 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.569482    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.575737 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.575622    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.630916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.630880    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btr6j\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-kube-api-access-btr6j\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.632425 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.632396    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.869059 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.869029    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:13.944222 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.942339    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 12:35:13.944222 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:13.942390    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 12:35:14.327398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.327291    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"]
Nov 01 12:35:14.342724 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:35:14.342691    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda17bdbf6_1545_43fd_8053_92a61f75edc1.slice/crio-6f93621a4343554e090603cdc746a33a1c5bac02b4c3719ebffd93b5be2e8e93 WatchSource:0}: Error finding container 6f93621a4343554e090603cdc746a33a1c5bac02b4c3719ebffd93b5be2e8e93: Status 404 returned error can't find the container with id 6f93621a4343554e090603cdc746a33a1c5bac02b4c3719ebffd93b5be2e8e93
Nov 01 12:35:14.345204 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.345172    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:14.345297 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.345240    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:14.345297 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.345282    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:14.734098 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.734058    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" event={"ID":"a17bdbf6-1545-43fd-8053-92a61f75edc1","Type":"ContainerStarted","Data":"098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0"}
Nov 01 12:35:14.734310 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.734109    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" event={"ID":"a17bdbf6-1545-43fd-8053-92a61f75edc1","Type":"ContainerStarted","Data":"6f93621a4343554e090603cdc746a33a1c5bac02b4c3719ebffd93b5be2e8e93"}
Nov 01 12:35:14.847284 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.847234    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" podStartSLOduration=1.847219221 podStartE2EDuration="1.847219221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:14.843660606 +0000 UTC m=+774.761456308" watchObservedRunningTime="2026-11-01 12:35:14.847219221 +0000 UTC m=+774.765014886"
Nov 01 12:35:14.869137 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.869110    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:14.942936 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.942478    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 12:35:14.942936 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:14.942542    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 12:35:15.546005 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:15.545079    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:35:15.871661 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:15.871627    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 12:35:15.872089 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:15.871675    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 12:35:15.950998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:15.945409    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 12:35:15.950998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:15.945474    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 12:35:16.869828 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:16.869791    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 12:35:16.870043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:16.869842    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 12:35:16.952691 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:16.952549    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 01 12:35:16.952691 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:16.952611    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 01 12:35:17.869610 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:17.869570    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 12:35:17.869785 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:17.869639    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 12:35:17.946601 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:17.946573    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:35:17.947585 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:17.947567    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:35:18.869652 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:18.869610    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 12:35:18.870192 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:18.869675    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 12:35:19.137562 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:35:19.137474    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:35:19.741229 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.741195    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"]
Nov 01 12:35:19.751273 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.751231    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:19.795578 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.795531    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"]
Nov 01 12:35:19.887265 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.887231    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-ecd640-serving-cert\""
Nov 01 12:35:19.889499 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.888465    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Nov 01 12:35:19.889499 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.889166    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Nov 01 12:35:19.988994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.988634    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665bb20c-fefd-4732-be3b-553c13774140-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:19.988994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.988732    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgkk9\" (UniqueName: \"kubernetes.io/projected/665bb20c-fefd-4732-be3b-553c13774140-kube-api-access-tgkk9\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:19.988994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.988780    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-metrics-certs\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:19.988994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:19.988858    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.090052 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.089939    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgkk9\" (UniqueName: \"kubernetes.io/projected/665bb20c-fefd-4732-be3b-553c13774140-kube-api-access-tgkk9\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.090246 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.090060    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-metrics-certs\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.090246 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.090157    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.090370 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.090252    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665bb20c-fefd-4732-be3b-553c13774140-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.091271 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.091242    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665bb20c-fefd-4732-be3b-553c13774140-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.093713 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.093686    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-metrics-certs\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.093829 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.093686    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-maas-api-tls\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.166007 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.161634    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgkk9\" (UniqueName: \"kubernetes.io/projected/665bb20c-fefd-4732-be3b-553c13774140-kube-api-access-tgkk9\") pod \"maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.396049 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.396012    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:20.873939 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.873910    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:20.874548 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.874431    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:20.875007 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.874989    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:35:20.882446 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:20.882426    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"]
Nov 01 12:35:20.883743 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:35:20.883715    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665bb20c_fefd_4732_be3b_553c13774140.slice/crio-6598fdcb18d1b637cc65b860b517de7fd768df1f14a249e1b22499270efe4d5c WatchSource:0}: Error finding container 6598fdcb18d1b637cc65b860b517de7fd768df1f14a249e1b22499270efe4d5c: Status 404 returned error can't find the container with id 6598fdcb18d1b637cc65b860b517de7fd768df1f14a249e1b22499270efe4d5c
Nov 01 12:35:21.791627 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:21.791590    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g" event={"ID":"665bb20c-fefd-4732-be3b-553c13774140","Type":"ContainerStarted","Data":"6598fdcb18d1b637cc65b860b517de7fd768df1f14a249e1b22499270efe4d5c"}
Nov 01 12:35:22.800320 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:22.800284    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g" event={"ID":"665bb20c-fefd-4732-be3b-553c13774140","Type":"ContainerStarted","Data":"7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0"}
Nov 01 12:35:22.800724 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:22.800403    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:22.928598 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:22.928527    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g" podStartSLOduration=2.960381945 podStartE2EDuration="3.928493212s" totalImagesPullingTime="968.111267ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:22.926156221 +0000 UTC m=+782.843951888" watchObservedRunningTime="2026-11-01 12:35:22.928493212 +0000 UTC m=+782.846288878"
Nov 01 12:35:28.895045 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.895013    4095 generic.go:356] "Generic (PLEG): container finished" podID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerID="22236c497292401b9ca72a58439df4ddc2238762a796794ded066bc01690895c" exitCode=0
Nov 01 12:35:28.895489 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.895060    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" event={"ID":"94d15666-ddd7-4270-8e31-4630a4b89b39","Type":"ContainerDied","Data":"22236c497292401b9ca72a58439df4ddc2238762a796794ded066bc01690895c"}
Nov 01 12:35:28.895489 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.895485    4095 scope.go:122] "RemoveContainer" containerID="22236c497292401b9ca72a58439df4ddc2238762a796794ded066bc01690895c"
Nov 01 12:35:28.896809 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.896787    4095 generic.go:356] "Generic (PLEG): container finished" podID="09815128-1266-4fed-b6a1-d059cc352fc4" containerID="198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756" exitCode=0
Nov 01 12:35:28.896904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.896825    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" event={"ID":"09815128-1266-4fed-b6a1-d059cc352fc4","Type":"ContainerDied","Data":"198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756"}
Nov 01 12:35:28.896945 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.896904    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:28.897022 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.896991    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:28.897072 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.897036    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:28.897155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.897141    4095 scope.go:122] "RemoveContainer" containerID="198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756"
Nov 01 12:35:28.898322 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.898298    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:28.898400 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.898357    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:28.898440 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:28.898403    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:35:29.592564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.592531    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"]
Nov 01 12:35:29.596507 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.596488    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.647995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.647603    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-metrics-certs\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.647995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.647732    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh7cb\" (UniqueName: \"kubernetes.io/projected/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-kube-api-access-mh7cb\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.647995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.647781    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.647995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.647858    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.657186 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.657157    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-9514b0-serving-cert\""
Nov 01 12:35:29.657782 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.657581    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"]
Nov 01 12:35:29.759065 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.757709    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh7cb\" (UniqueName: \"kubernetes.io/projected/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-kube-api-access-mh7cb\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.759065 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.757786    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.759065 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.757875    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.759065 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.757965    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-metrics-certs\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.759065 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:35:29.758217    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-serving-cert: secret "maas-api-e2e-worker-w5-9514b0-serving-cert" not found
Nov 01 12:35:29.759065 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:35:29.758283    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls podName:18010a26-70d2-46fb-a44e-3bedc5c1d6b1 nodeName:}" failed. No retries permitted until 2026-11-01 12:35:30.25826353 +0000 UTC m=+790.176059188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls") pod "maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm" (UID: "18010a26-70d2-46fb-a44e-3bedc5c1d6b1") : secret "maas-api-e2e-worker-w5-9514b0-serving-cert" not found
Nov 01 12:35:29.759722 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.759314    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.766500 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.766475    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-metrics-certs\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.852926 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.852667    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh7cb\" (UniqueName: \"kubernetes.io/projected/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-kube-api-access-mh7cb\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:29.869094 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.868916    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 12:35:29.927321 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.927284    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" event={"ID":"09815128-1266-4fed-b6a1-d059cc352fc4","Type":"ContainerStarted","Data":"ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe"}
Nov 01 12:35:29.958229 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:29.958193    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" event={"ID":"94d15666-ddd7-4270-8e31-4630a4b89b39","Type":"ContainerStarted","Data":"df25cfc45df80befabf5d5a7a9d9f3beb798783553b48f244883453f7d6b21f8"}
Nov 01 12:35:30.265793 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.265750    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:30.268510 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.268483    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls\") pod \"maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:30.522340 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.520416    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:35:30.522627 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.522599    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 12:35:30.522825 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.522798    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 12:35:30.528340 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.527157    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:30.553028 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.552666    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:35:30.553814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.553790    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 12:35:30.554031 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.554006    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 12:35:30.954128 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.954096    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"]
Nov 01 12:35:30.964158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:30.964132    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm" event={"ID":"18010a26-70d2-46fb-a44e-3bedc5c1d6b1","Type":"ContainerStarted","Data":"4e5f3f2d0c1bbf455a7f481ac0fbac3e1b1cf9a5dfc1901eec0078a40ad63dd3"}
Nov 01 12:35:31.521995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:31.519444    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 12:35:31.521995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:31.519503    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 12:35:31.539994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:31.537340    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 12:35:31.539994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:31.537398    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 12:35:31.542998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:31.542063    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:35:31.542998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:31.542223    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:35:32.520996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:32.520358    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 12:35:32.520996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:32.520423    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 12:35:32.542006 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:32.538319    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 12:35:32.542006 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:32.538371    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 12:35:32.977326 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:32.977289    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm" event={"ID":"18010a26-70d2-46fb-a44e-3bedc5c1d6b1","Type":"ContainerStarted","Data":"30775287d46fe41030475017bb0cbf2126197d3bd3ca31973110850c2d4829a7"}
Nov 01 12:35:32.977498 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:32.977456    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:33.090321 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:33.090257    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm" podStartSLOduration=3.073208863 podStartE2EDuration="4.090237017s" totalImagesPullingTime="1.017028154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:35:33.087394258 +0000 UTC m=+793.005189925" watchObservedRunningTime="2026-11-01 12:35:33.090237017 +0000 UTC m=+793.008032684"
Nov 01 12:35:33.539992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:33.529106    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 12:35:33.539992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:33.529187    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 12:35:33.553017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:33.545096    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 12:35:33.553017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:33.545170    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 12:35:34.523996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:34.522381    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 01 12:35:34.523996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:34.522443    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 01 12:35:34.544994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:34.544361    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 01 12:35:34.544994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:34.544419    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 01 12:35:35.524302 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:35.524269    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:35:35.525208 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:35.525188    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:35:35.540134 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:35.540115    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:35:35.540921 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:35.540907    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 12:35:38.990902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:38.990875    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 12:35:49.113431 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:35:49.113395    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:35:58.386741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.386709    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"]
Nov 01 12:35:58.391890 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.391867    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.438851 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.438811    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8vfg\" (UniqueName: \"kubernetes.io/projected/a290f39b-7f4c-4927-8fa3-3094bb02374c-kube-api-access-z8vfg\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.439043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.438920    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-model-cache\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.439127 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.439109    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-dshm\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.439182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.439171    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a290f39b-7f4c-4927-8fa3-3094bb02374c-tls-certs\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.439345 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.439321    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-tmp-dir\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.439432 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.439417    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-home\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.464458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.464428    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"]
Nov 01 12:35:58.489603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.489566    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"kube-root-ca.crt\""
Nov 01 12:35:58.489764 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.489609    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"openshift-service-ca.crt\""
Nov 01 12:35:58.490126 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.490099    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"default-dockercfg-8rwpt\""
Nov 01 12:35:58.490300 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.490113    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"simulator-w2-c7206f-kserve-self-signed-certs\""
Nov 01 12:35:58.540612 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.540578    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8vfg\" (UniqueName: \"kubernetes.io/projected/a290f39b-7f4c-4927-8fa3-3094bb02374c-kube-api-access-z8vfg\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.540834 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.540629    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-model-cache\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.540834 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.540715    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-dshm\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.540834 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.540758    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a290f39b-7f4c-4927-8fa3-3094bb02374c-tls-certs\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.540834 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.540804    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-tmp-dir\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.541112 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.540857    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-home\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.541426 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.541400    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-home\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.542011 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.541965    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-model-cache\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.542272 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.542244    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-tmp-dir\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.545906 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.545878    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-dshm\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.546136 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.546111    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a290f39b-7f4c-4927-8fa3-3094bb02374c-tls-certs\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.595902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.595865    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8vfg\" (UniqueName: \"kubernetes.io/projected/a290f39b-7f4c-4927-8fa3-3094bb02374c-kube-api-access-z8vfg\") pod \"simulator-w2-c7206f-kserve-849f5969b5-2n59s\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") " pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:58.704020 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:58.703912    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:35:59.073511 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:59.073490    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"]
Nov 01 12:35:59.074475 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:35:59.074447    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda290f39b_7f4c_4927_8fa3_3094bb02374c.slice/crio-1941f5922c319750128627fc576b25943d853917ac94ebd0d3c76836f55378d9 WatchSource:0}: Error finding container 1941f5922c319750128627fc576b25943d853917ac94ebd0d3c76836f55378d9: Status 404 returned error can't find the container with id 1941f5922c319750128627fc576b25943d853917ac94ebd0d3c76836f55378d9
Nov 01 12:35:59.132285 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:59.132258    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" event={"ID":"a290f39b-7f4c-4927-8fa3-3094bb02374c","Type":"ContainerStarted","Data":"1941f5922c319750128627fc576b25943d853917ac94ebd0d3c76836f55378d9"}
Nov 01 12:35:59.988439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:59.988410    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"]
Nov 01 12:35:59.992439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:35:59.992421    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.041691 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.041652    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f7427d-openshift-default-dockercfg-zdm4c\""
Nov 01 12:36:00.060626 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.060596    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.060789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.060669    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.060789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.060744    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.060910 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.060827    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.060910 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.060849    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d753619-fbf8-4087-85ca-e2a61d971e8d-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.061047 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.060988    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8rgt\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-kube-api-access-b8rgt\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.061047 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.061018    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.061047 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.061038    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.061192 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.061077    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.082869 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.082839    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"]
Nov 01 12:36:00.141074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.141019    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" event={"ID":"a290f39b-7f4c-4927-8fa3-3094bb02374c","Type":"ContainerStarted","Data":"4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490"}
Nov 01 12:36:00.162209 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162177    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162374 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162270    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162374 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162290    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162374 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162316    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162374 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162339    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d753619-fbf8-4087-85ca-e2a61d971e8d-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162404    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8rgt\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-kube-api-access-b8rgt\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162427    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162456    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162486    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162822 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162630    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.162822 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.162686    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.163139 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.163116    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.163212 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.163119    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.163486 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.163469    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d753619-fbf8-4087-85ca-e2a61d971e8d-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.164698 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.164674    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.164808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.164764    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.219554 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.219521    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.219764 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.219740    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8rgt\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-kube-api-access-b8rgt\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.249818 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.249732    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podStartSLOduration=2.249719726 podStartE2EDuration="2.249719726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:00.248384776 +0000 UTC m=+820.166180444" watchObservedRunningTime="2026-11-01 12:36:00.249719726 +0000 UTC m=+820.167515392"
Nov 01 12:36:00.305819 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.305787    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:00.704440 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.704407    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:36:00.723153 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:36:00.723118    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d753619_fbf8_4087_85ca_e2a61d971e8d.slice/crio-e6eccd67bc8863ab61226db74f98fbae8e8c54c430c46ab805c00e3ff5b5b613 WatchSource:0}: Error finding container e6eccd67bc8863ab61226db74f98fbae8e8c54c430c46ab805c00e3ff5b5b613: Status 404 returned error can't find the container with id e6eccd67bc8863ab61226db74f98fbae8e8c54c430c46ab805c00e3ff5b5b613
Nov 01 12:36:00.723810 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.723788    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"]
Nov 01 12:36:00.725658 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.725628    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:36:00.725746 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.725692    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:36:00.725746 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:00.725722    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:36:01.148630 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:01.148594    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" event={"ID":"0d753619-fbf8-4087-85ca-e2a61d971e8d","Type":"ContainerStarted","Data":"220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a"}
Nov 01 12:36:01.149049 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:01.148660    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" event={"ID":"0d753619-fbf8-4087-85ca-e2a61d971e8d","Type":"ContainerStarted","Data":"e6eccd67bc8863ab61226db74f98fbae8e8c54c430c46ab805c00e3ff5b5b613"}
Nov 01 12:36:01.260776 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:01.260724    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" podStartSLOduration=2.260709343 podStartE2EDuration="2.260709343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:01.260100466 +0000 UTC m=+821.177896124" watchObservedRunningTime="2026-11-01 12:36:01.260709343 +0000 UTC m=+821.178505009"
Nov 01 12:36:01.317480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:01.317019    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:01.317480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:01.317156    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 12:36:01.317480 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:01.317204    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 12:36:02.322481 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:02.322437    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 12:36:02.323156 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:02.323122    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 12:36:03.313840 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:03.313369    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 12:36:03.313840 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:03.313450    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 12:36:04.311888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:04.311853    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 12:36:04.312504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:04.312472    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 12:36:05.315993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:05.306260    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 12:36:05.315993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:05.306330    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 12:36:06.309002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:06.306312    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused" start-of-body=
Nov 01 12:36:06.309002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:06.306375    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.50:15021/healthz/ready\": dial tcp 10.134.0.50:15021: connect: connection refused"
Nov 01 12:36:07.310665 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:07.310637    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:07.311126 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:07.310914    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:07.311651 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:07.311635    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 12:36:08.705061 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:08.705014    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:36:08.717838 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:08.717813    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:36:09.206767 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:09.206738    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:36:14.033679 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.033645    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"]
Nov 01 12:36:14.038462 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.038441    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.068309 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.068228    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.068309 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.068304    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.068536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.068387    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q2fj\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-kube-api-access-5q2fj\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.068536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.068458    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/030c9f49-c520-4f1b-b94e-fac2efd057b7-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.068641 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.068572    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.068703 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.068658    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.068774 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.068753    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.068880 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.068850    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.068937 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.068886    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.138357 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.138328    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"]
Nov 01 12:36:14.170646 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.170600    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/030c9f49-c520-4f1b-b94e-fac2efd057b7-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.170816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.170658    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.170816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.170718    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.170938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.170811    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171025 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171001    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171050    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171144 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171111    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171200 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171173    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171256 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171204    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q2fj\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-kube-api-access-5q2fj\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171441 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171416    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/030c9f49-c520-4f1b-b94e-fac2efd057b7-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171636 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171615    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171801 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171780    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171882 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171830    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.171882 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.171868    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.173591 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.173568    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.173787 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.173769    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.225820 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.225790    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q2fj\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-kube-api-access-5q2fj\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.225997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.225794    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.351965 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.351864    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:14.769683 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.769653    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"]
Nov 01 12:36:14.770203 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:36:14.770181    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod030c9f49_c520_4f1b_b94e_fac2efd057b7.slice/crio-afcf8e372dcbe6df5fc57bc58eaeb59f460a0812b7cf339b6e3ec4a9992b96c6 WatchSource:0}: Error finding container afcf8e372dcbe6df5fc57bc58eaeb59f460a0812b7cf339b6e3ec4a9992b96c6: Status 404 returned error can't find the container with id afcf8e372dcbe6df5fc57bc58eaeb59f460a0812b7cf339b6e3ec4a9992b96c6
Nov 01 12:36:14.772628 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.772596    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:36:14.772711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.772659    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:36:14.772711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:14.772689    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:36:15.230584 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:15.230546    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" event={"ID":"030c9f49-c520-4f1b-b94e-fac2efd057b7","Type":"ContainerStarted","Data":"dc3b32a4b5a02fe62e6bc9ec74a31281826159d4f85e380f6f7261e7a05d2ad9"}
Nov 01 12:36:15.231007 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:15.230609    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" event={"ID":"030c9f49-c520-4f1b-b94e-fac2efd057b7","Type":"ContainerStarted","Data":"afcf8e372dcbe6df5fc57bc58eaeb59f460a0812b7cf339b6e3ec4a9992b96c6"}
Nov 01 12:36:15.342423 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:15.342358    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" podStartSLOduration=2.342336869 podStartE2EDuration="2.342336869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:36:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:15.341652344 +0000 UTC m=+835.259448007" watchObservedRunningTime="2026-11-01 12:36:15.342336869 +0000 UTC m=+835.260132536"
Nov 01 12:36:15.352436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:15.352404    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:15.353899 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:15.353868    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 12:36:15.354060 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:15.353922    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 12:36:16.352535 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:16.352499    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 12:36:16.352914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:16.352570    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 12:36:17.352635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:17.352558    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 12:36:17.353030 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:17.352633    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 12:36:18.352796 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:18.352760    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 12:36:18.353177 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:18.352810    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 12:36:19.136602 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:36:19.136566    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:36:19.352992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:19.352947    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 12:36:19.353384 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:19.353016    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 12:36:20.352449 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:20.352415    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Nov 01 12:36:20.352628 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:20.352466    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Nov 01 12:36:21.357278 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:21.357242    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:21.357719 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:21.357540    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:21.358200 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:21.358182    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:36:29.813953 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.813917    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"]
Nov 01 12:36:29.819024 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.819004    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:29.916342 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.916312    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"kube-root-ca.crt\""
Nov 01 12:36:29.916511 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.916353    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"openshift-service-ca.crt\""
Nov 01 12:36:29.916585 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.916565    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"premium-w0-94c94b-kserve-self-signed-certs\""
Nov 01 12:36:29.916585 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.916581    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"default-dockercfg-8xv8p\""
Nov 01 12:36:29.918642 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.918621    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"]
Nov 01 12:36:29.972879 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.972842    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-model-cache\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:29.973235 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.973210    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tls-certs\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:29.973377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.973350    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-dshm\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:29.973556 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.973408    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxm6x\" (UniqueName: \"kubernetes.io/projected/73c3bfd2-3551-4ea5-8a5b-98379b29796a-kube-api-access-rxm6x\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:29.973556 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.973436    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tmp-dir\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:29.973556 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:29.973497    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-home\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.075079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.075002    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-home\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.075252 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.075120    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-model-cache\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.075252 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.075186    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tls-certs\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.075252 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.075215    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-dshm\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.075419 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.075254    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxm6x\" (UniqueName: \"kubernetes.io/projected/73c3bfd2-3551-4ea5-8a5b-98379b29796a-kube-api-access-rxm6x\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.075419 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.075286    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tmp-dir\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.075534 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.075435    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-home\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.075534 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.075500    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-model-cache\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.075689 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.075671    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tmp-dir\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.077360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.077338    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-dshm\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.077776 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.077757    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tls-certs\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.129834 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.129802    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxm6x\" (UniqueName: \"kubernetes.io/projected/73c3bfd2-3551-4ea5-8a5b-98379b29796a-kube-api-access-rxm6x\") pod \"premium-w0-94c94b-kserve-68cf4bb599-k5mvd\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") " pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.156210 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.156183    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:30.547360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:30.547334    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"]
Nov 01 12:36:30.549281 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:36:30.549253    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73c3bfd2_3551_4ea5_8a5b_98379b29796a.slice/crio-12a51b66fc12bd3b3cc915f3be4e45cf2ba01588ea1c9a88ac2330371a94d75a WatchSource:0}: Error finding container 12a51b66fc12bd3b3cc915f3be4e45cf2ba01588ea1c9a88ac2330371a94d75a: Status 404 returned error can't find the container with id 12a51b66fc12bd3b3cc915f3be4e45cf2ba01588ea1c9a88ac2330371a94d75a
Nov 01 12:36:31.318348 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:31.318255    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" event={"ID":"73c3bfd2-3551-4ea5-8a5b-98379b29796a","Type":"ContainerStarted","Data":"27b0458049f6d9583d6131d12570b9dfc4e3c8cfd78ad5dc2823ea6570a78a91"}
Nov 01 12:36:31.318348 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:31.318301    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" event={"ID":"73c3bfd2-3551-4ea5-8a5b-98379b29796a","Type":"ContainerStarted","Data":"12a51b66fc12bd3b3cc915f3be4e45cf2ba01588ea1c9a88ac2330371a94d75a"}
Nov 01 12:36:31.429404 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:31.429307    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podStartSLOduration=2.429290852 podStartE2EDuration="2.429290852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:36:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:36:31.426771778 +0000 UTC m=+851.344567444" watchObservedRunningTime="2026-11-01 12:36:31.429290852 +0000 UTC m=+851.347086519"
Nov 01 12:36:32.156831 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:32.156797    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:40.157073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:40.157038    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:40.169467 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:40.169443    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:40.384304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:36:40.384278    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:36:49.113955 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:36:49.113913    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:37:02.160884 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.160854    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"]
Nov 01 12:37:02.165773 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.165756    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.262982 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.262948    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"premium-w2-c7206f-kserve-self-signed-certs\""
Nov 01 12:37:02.265452 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.265430    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"]
Nov 01 12:37:02.269521 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.269499    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-home\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.269622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.269555    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-dshm\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.269676 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.269643    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-456fh\" (UniqueName: \"kubernetes.io/projected/54698a98-165c-4f90-a571-6fee68c2d0f1-kube-api-access-456fh\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.269717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.269691    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-tmp-dir\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.269788 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.269774    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-model-cache\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.269870 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.269860    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54698a98-165c-4f90-a571-6fee68c2d0f1-tls-certs\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.370566 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.370533    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54698a98-165c-4f90-a571-6fee68c2d0f1-tls-certs\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.370742 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.370595    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-home\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.370742 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.370661    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-dshm\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.370742 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.370711    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-456fh\" (UniqueName: \"kubernetes.io/projected/54698a98-165c-4f90-a571-6fee68c2d0f1-kube-api-access-456fh\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.370742 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.370740    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-tmp-dir\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.370954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.370867    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-model-cache\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.371121 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.371100    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-tmp-dir\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.371206 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.371181    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-home\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.371365 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.371348    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-model-cache\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.372927 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.372908    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-dshm\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.373200 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.373183    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54698a98-165c-4f90-a571-6fee68c2d0f1-tls-certs\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.424683 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.424619    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-456fh\" (UniqueName: \"kubernetes.io/projected/54698a98-165c-4f90-a571-6fee68c2d0f1-kube-api-access-456fh\") pod \"premium-w2-c7206f-kserve-75bd596d8-dkvv4\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") " pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.479227 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.479195    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:02.868090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:02.868064    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"]
Nov 01 12:37:02.869562 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:37:02.869536    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod54698a98_165c_4f90_a571_6fee68c2d0f1.slice/crio-c1df9a5a077b164d473793824805fb0b474f4c9bc7e733769c7f33bca50e4afd WatchSource:0}: Error finding container c1df9a5a077b164d473793824805fb0b474f4c9bc7e733769c7f33bca50e4afd: Status 404 returned error can't find the container with id c1df9a5a077b164d473793824805fb0b474f4c9bc7e733769c7f33bca50e4afd
Nov 01 12:37:03.554208 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:03.554175    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" event={"ID":"54698a98-165c-4f90-a571-6fee68c2d0f1","Type":"ContainerStarted","Data":"37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243"}
Nov 01 12:37:03.554208 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:03.554215    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" event={"ID":"54698a98-165c-4f90-a571-6fee68c2d0f1","Type":"ContainerStarted","Data":"c1df9a5a077b164d473793824805fb0b474f4c9bc7e733769c7f33bca50e4afd"}
Nov 01 12:37:03.665666 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:03.665613    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podStartSLOduration=1.665599171 podStartE2EDuration="1.665599171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:37:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:37:03.66279276 +0000 UTC m=+883.580588426" watchObservedRunningTime="2026-11-01 12:37:03.665599171 +0000 UTC m=+883.583394886"
Nov 01 12:37:04.479606 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:04.479572    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:12.479929 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:12.479894    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:12.492634 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:12.492614    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:12.614571 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:12.614538    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:37:19.117446 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:37:19.117404    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:37:20.258734 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:20.258705    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:37:20.262278 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:20.262255    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:37:35.835366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.835328    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"]
Nov 01 12:37:35.839329 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.839307    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.849351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.849332    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tls-certs\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.849456 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.849359    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nmbb\" (UniqueName: \"kubernetes.io/projected/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-kube-api-access-2nmbb\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.849456 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.849384    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-home\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.849545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.849458    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tmp-dir\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.849584 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.849547    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-dshm\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.849627 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.849604    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-model-cache\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.937479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.937436    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"default-dockercfg-l9jz2\""
Nov 01 12:37:35.937655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.937542    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"openshift-service-ca.crt\""
Nov 01 12:37:35.937655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.937563    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"premium-w4-06fd6b-kserve-self-signed-certs\""
Nov 01 12:37:35.937655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.937582    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"kube-root-ca.crt\""
Nov 01 12:37:35.940161 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.940139    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"]
Nov 01 12:37:35.950396 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.950376    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tmp-dir\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.950484 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.950427    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-dshm\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.950544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.950483    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-model-cache\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.950602 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.950581    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tls-certs\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.950660 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.950607    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nmbb\" (UniqueName: \"kubernetes.io/projected/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-kube-api-access-2nmbb\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.950660 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.950638    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-home\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.950848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.950828    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tmp-dir\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.950966 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.950948    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-model-cache\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.951050 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.951031    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-home\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.952645 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.952627    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-dshm\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:35.953023 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:35.953005    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tls-certs\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:36.008696 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:36.008661    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nmbb\" (UniqueName: \"kubernetes.io/projected/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-kube-api-access-2nmbb\") pod \"premium-w4-06fd6b-kserve-569c479d6-nr96l\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") " pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:36.150550 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:36.150516    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:36.557755 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:36.557726    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"]
Nov 01 12:37:36.558916 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:37:36.558885    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podba3a9abf_9e99_4e4d_832c_bcdb3c8673e7.slice/crio-c5b4a62fca6bc25795d56d746db3bdd9693990d104ec22631b4b32202037dc65 WatchSource:0}: Error finding container c5b4a62fca6bc25795d56d746db3bdd9693990d104ec22631b4b32202037dc65: Status 404 returned error can't find the container with id c5b4a62fca6bc25795d56d746db3bdd9693990d104ec22631b4b32202037dc65
Nov 01 12:37:36.754705 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:36.754657    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" event={"ID":"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7","Type":"ContainerStarted","Data":"5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c"}
Nov 01 12:37:36.754878 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:36.754716    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" event={"ID":"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7","Type":"ContainerStarted","Data":"c5b4a62fca6bc25795d56d746db3bdd9693990d104ec22631b4b32202037dc65"}
Nov 01 12:37:36.865542 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:36.865493    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podStartSLOduration=1.865477504 podStartE2EDuration="1.865477504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:37:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:37:36.862756541 +0000 UTC m=+916.780552206" watchObservedRunningTime="2026-11-01 12:37:36.865477504 +0000 UTC m=+916.783273170"
Nov 01 12:37:37.150950 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:37.150863    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:46.151610 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:46.151578    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:46.164901 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:46.164875    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:46.825415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:37:46.825381    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:37:49.114336 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:37:49.114300    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:38:19.151691 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:38:19.151645    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:38:26.809448 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.809411    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"]
Nov 01 12:38:26.813574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.813552    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.864466 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.864427    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-home\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.864829 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.864796    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c552b5a6-2b9c-4584-9e8a-d709799b5730-tls-certs\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.864945 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.864852    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-tmp-dir\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.864945 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.864878    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-dshm\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.865091 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.864951    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmrw8\" (UniqueName: \"kubernetes.io/projected/c552b5a6-2b9c-4584-9e8a-d709799b5730-kube-api-access-qmrw8\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.865282 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.865254    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-model-cache\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.911381 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.911347    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"openshift-service-ca.crt\""
Nov 01 12:38:26.911547 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.911524    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"kube-root-ca.crt\""
Nov 01 12:38:26.911627 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.911553    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"default-dockercfg-wsxnm\""
Nov 01 12:38:26.911627 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.911556    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-f7427d\"/\"premium-w6-f7427d-kserve-self-signed-certs\""
Nov 01 12:38:26.914172 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.914152    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"]
Nov 01 12:38:26.966131 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.966092    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmrw8\" (UniqueName: \"kubernetes.io/projected/c552b5a6-2b9c-4584-9e8a-d709799b5730-kube-api-access-qmrw8\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.968082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.967733    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-model-cache\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.968082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.967794    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-home\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.968082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.967888    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c552b5a6-2b9c-4584-9e8a-d709799b5730-tls-certs\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.968082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.967925    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-tmp-dir\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.968082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.967948    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-dshm\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.968473 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.968428    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-home\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.969549 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.968687    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-model-cache\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.969700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.969666    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-tmp-dir\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.972502 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.972458    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c552b5a6-2b9c-4584-9e8a-d709799b5730-tls-certs\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:26.974797 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:26.974772    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-dshm\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:27.021163 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:27.021137    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmrw8\" (UniqueName: \"kubernetes.io/projected/c552b5a6-2b9c-4584-9e8a-d709799b5730-kube-api-access-qmrw8\") pod \"premium-w6-f7427d-kserve-6c45dc7ccd-7mntw\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") " pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:27.126891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:27.126862    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:27.519047 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:27.519015    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"]
Nov 01 12:38:27.519841 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:38:27.519816    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc552b5a6_2b9c_4584_9e8a_d709799b5730.slice/crio-4ee43a5bb6fac83833076f9b4c1ea12ea824e6a4ecc0ad26dd72d2fefef3853e WatchSource:0}: Error finding container 4ee43a5bb6fac83833076f9b4c1ea12ea824e6a4ecc0ad26dd72d2fefef3853e: Status 404 returned error can't find the container with id 4ee43a5bb6fac83833076f9b4c1ea12ea824e6a4ecc0ad26dd72d2fefef3853e
Nov 01 12:38:28.052420 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:28.052390    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" event={"ID":"c552b5a6-2b9c-4584-9e8a-d709799b5730","Type":"ContainerStarted","Data":"a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e"}
Nov 01 12:38:28.052799 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:28.052428    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" event={"ID":"c552b5a6-2b9c-4584-9e8a-d709799b5730","Type":"ContainerStarted","Data":"4ee43a5bb6fac83833076f9b4c1ea12ea824e6a4ecc0ad26dd72d2fefef3853e"}
Nov 01 12:38:28.127710 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:28.127681    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:28.161051 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:28.160996    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podStartSLOduration=2.160957651 podStartE2EDuration="2.160957651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:38:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:38:28.159778277 +0000 UTC m=+968.077573945" watchObservedRunningTime="2026-11-01 12:38:28.160957651 +0000 UTC m=+968.078753350"
Nov 01 12:38:37.127035 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:37.127001    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:37.143096 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:37.143056    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:38.147712 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:38:38.147684    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:38:49.113287 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:38:49.113245    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:39:19.116134 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:39:19.116095    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:39:43.798423 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.798343    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"]
Nov 01 12:39:43.801892 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.801876    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:43.900999 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.900959    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-c7206f\"/\"distinct-w2-c7206f-kserve-self-signed-certs\""
Nov 01 12:39:43.903632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.903607    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"]
Nov 01 12:39:43.978048 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.978019    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-model-cache\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:43.978219 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.978142    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgcs5\" (UniqueName: \"kubernetes.io/projected/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-kube-api-access-lgcs5\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:43.978219 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.978206    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-home\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:43.978343 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.978327    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tls-certs\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:43.978379 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.978359    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-dshm\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:43.978425 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:43.978378    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tmp-dir\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.079448 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.079368    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tls-certs\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.079448 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.079423    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-dshm\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.079680 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.079465    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tmp-dir\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.079680 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.079561    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-model-cache\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.079680 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.079663    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgcs5\" (UniqueName: \"kubernetes.io/projected/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-kube-api-access-lgcs5\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.079822 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.079736    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-home\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.079966 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.079941    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tmp-dir\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.080069 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.080003    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-model-cache\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.080174 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.080155    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-home\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.083307 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.083279    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-dshm\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.083544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.083528    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tls-certs\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.133301 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.133274    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgcs5\" (UniqueName: \"kubernetes.io/projected/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-kube-api-access-lgcs5\") pod \"distinct-w2-c7206f-kserve-58fd8bf9-lwczv\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") " pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.301148 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.301116    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"]
Nov 01 12:39:44.305367 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.305349    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.360326 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.360301    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"]
Nov 01 12:39:44.414141 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.414110    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:44.486889 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.486852    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.487029 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.486931    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.487077 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.487030    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mgdq\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-kube-api-access-6mgdq\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.487077 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.487068    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.487147 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.487135    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.487196 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.487182    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.487248 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.487234    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.487347 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.487334    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.487394 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.487365    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588323    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588554 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588378    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588554 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588423    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588554 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588505    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588554 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588532    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588576    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588627    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588718    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mgdq\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-kube-api-access-6mgdq\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588759    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588960 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588802    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.588960 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.588885    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.589118 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.589075    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.589118 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.589097    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.589507 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.589480    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.590621 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.590596    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.591229 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.591212    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.642497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.642436    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.642497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.642484    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mgdq\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-kube-api-access-6mgdq\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:44.777610 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.777580    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"]
Nov 01 12:39:44.778423 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:39:44.778395    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e8cde83_0122_42ac_a2d5_e6c2e6d97703.slice/crio-e8a8922c2c183f76a64963114eb37833442c381ac18631cc051af08d1945f397 WatchSource:0}: Error finding container e8a8922c2c183f76a64963114eb37833442c381ac18631cc051af08d1945f397: Status 404 returned error can't find the container with id e8a8922c2c183f76a64963114eb37833442c381ac18631cc051af08d1945f397
Nov 01 12:39:44.922742 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:44.922642    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:45.378438 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.378411    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"]
Nov 01 12:39:45.379495 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:39:45.379467    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0490afc7_ea75_4d2c_b87f_ad207c89dab2.slice/crio-fc4b4289478e79ca41a2db2ff8cae29311bd9c6cbb97387dd169b765e23b2ca6 WatchSource:0}: Error finding container fc4b4289478e79ca41a2db2ff8cae29311bd9c6cbb97387dd169b765e23b2ca6: Status 404 returned error can't find the container with id fc4b4289478e79ca41a2db2ff8cae29311bd9c6cbb97387dd169b765e23b2ca6
Nov 01 12:39:45.382668 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.382637    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:39:45.382794 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.382720    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:39:45.382794 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.382775    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:39:45.602000 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.601931    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" event={"ID":"3e8cde83-0122-42ac-a2d5-e6c2e6d97703","Type":"ContainerStarted","Data":"2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0"}
Nov 01 12:39:45.602000 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.601996    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" event={"ID":"3e8cde83-0122-42ac-a2d5-e6c2e6d97703","Type":"ContainerStarted","Data":"e8a8922c2c183f76a64963114eb37833442c381ac18631cc051af08d1945f397"}
Nov 01 12:39:45.603784 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.603758    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" event={"ID":"0490afc7-ea75-4d2c-b87f-ad207c89dab2","Type":"ContainerStarted","Data":"8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c"}
Nov 01 12:39:45.603911 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.603804    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" event={"ID":"0490afc7-ea75-4d2c-b87f-ad207c89dab2","Type":"ContainerStarted","Data":"fc4b4289478e79ca41a2db2ff8cae29311bd9c6cbb97387dd169b765e23b2ca6"}
Nov 01 12:39:45.714469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.714365    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podStartSLOduration=2.714350924 podStartE2EDuration="2.714350924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:45.709535764 +0000 UTC m=+1045.627331446" watchObservedRunningTime="2026-11-01 12:39:45.714350924 +0000 UTC m=+1045.632146587"
Nov 01 12:39:45.827878 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.827817    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" podStartSLOduration=1.827799303 podStartE2EDuration="1.827799303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:39:45.823919205 +0000 UTC m=+1045.741714872" watchObservedRunningTime="2026-11-01 12:39:45.827799303 +0000 UTC m=+1045.745594970"
Nov 01 12:39:45.924998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:45.924051    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:46.421995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:46.415072    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:46.928963 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:46.928917    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:39:46.929473 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:46.929014    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 12:39:47.924380 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:47.924345    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:39:47.924649 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:47.924620    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 12:39:48.927385 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:48.927349    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:39:48.927890 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:48.927413    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 12:39:49.138905 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:39:49.138859    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:39:49.924991 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:49.924347    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:39:49.924991 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:49.924404    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 12:39:50.923356 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:50.923314    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 01 12:39:50.923963 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:50.923931    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 01 12:39:52.169119 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:52.169056    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:52.170397 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:52.170370    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:52.171428 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:52.171374    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:39:54.414257 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:54.414224    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:54.426738 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:54.426715    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:54.688560 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:54.688484    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:39:59.529751 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.529721    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"]
Nov 01 12:39:59.534062 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.534044    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.597675 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.597641    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.597854 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.597681    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.597854 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.597782    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.597955 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.597890    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.597955 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.597917    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljgzm\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-kube-api-access-ljgzm\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.598076 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.597991    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.598076 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.598011    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.598076 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.598037    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973d4e-e991-4011-ac52-eeb345a4e4de-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.598174 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.598156    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699149 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699109    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699149 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699157    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljgzm\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-kube-api-access-ljgzm\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699193    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699214    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699246    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973d4e-e991-4011-ac52-eeb345a4e4de-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699329    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699433    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699468    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699532    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699563    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699608    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-certs\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699887 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699836    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-data\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.699930 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.699911    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-credential-socket\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.700103 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.700082    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973d4e-e991-4011-ac52-eeb345a4e4de-istiod-ca-cert\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.701692 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.701668    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-envoy\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.701782 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.701753    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-podinfo\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.723824 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.723783    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"]
Nov 01 12:39:59.753022 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.752961    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-token\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.753233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.753205    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljgzm\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-kube-api-access-ljgzm\") pod \"e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") " pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:39:59.847589 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:39:59.847506    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:40:00.263726 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:00.263700    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"]
Nov 01 12:40:00.264290 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:40:00.264263    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57973d4e_e991_4011_ac52_eeb345a4e4de.slice/crio-7f4486f5b18ded193631d7af2965d0980d382add4fa541d3a16eb5bb60ed1294 WatchSource:0}: Error finding container 7f4486f5b18ded193631d7af2965d0980d382add4fa541d3a16eb5bb60ed1294: Status 404 returned error can't find the container with id 7f4486f5b18ded193631d7af2965d0980d382add4fa541d3a16eb5bb60ed1294
Nov 01 12:40:00.266915 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:00.266883    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:40:00.267015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:00.266950    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:40:00.267015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:00.267002    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:40:00.727958 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:00.727928    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" event={"ID":"57973d4e-e991-4011-ac52-eeb345a4e4de","Type":"ContainerStarted","Data":"21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc"}
Nov 01 12:40:00.728349 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:00.727996    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" event={"ID":"57973d4e-e991-4011-ac52-eeb345a4e4de","Type":"ContainerStarted","Data":"7f4486f5b18ded193631d7af2965d0980d382add4fa541d3a16eb5bb60ed1294"}
Nov 01 12:40:00.848267 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:00.848229    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:40:01.859995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:01.859692    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 12:40:01.859995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:01.859746    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 12:40:02.848685 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:02.848655    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 12:40:02.848854 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:02.848706    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 12:40:03.848350 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:03.848316    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 12:40:03.848733 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:03.848368    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 12:40:04.848106 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:04.848074    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 12:40:04.848357 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:04.848154    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 12:40:05.848148 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:05.848117    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 12:40:05.848545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:05.848167    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 12:40:06.848146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:06.848115    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Nov 01 12:40:06.848548 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:06.848182    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Nov 01 12:40:08.055376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:08.055348    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:40:08.056156 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:08.055839    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:40:08.056654 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:08.056618    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:40:08.258887 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:08.258834    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podStartSLOduration=9.258820735 podStartE2EDuration="9.258820735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:00.887877605 +0000 UTC m=+1060.805673271" watchObservedRunningTime="2026-11-01 12:40:08.258820735 +0000 UTC m=+1068.176616400"
Nov 01 12:40:19.115833 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:40:19.115785    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:40:32.251333 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.251302    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"]
Nov 01 12:40:32.255290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.255275    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.354046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.354017    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"openshift-service-ca.crt\""
Nov 01 12:40:32.354046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.354042    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"default-dockercfg-vp2p9\""
Nov 01 12:40:32.354246 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.354081    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"distinct-w1-5b1bd4-kserve-self-signed-certs\""
Nov 01 12:40:32.354246 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.354114    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"kube-root-ca.crt\""
Nov 01 12:40:32.357050 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.357025    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"]
Nov 01 12:40:32.377786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.377758    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-model-cache\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.377926 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.377816    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-tmp-dir\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.377926 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.377884    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-home\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.377926 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.377920    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5c4201-96c1-40b9-be00-774ccc27e376-tls-certs\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.378065 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.378050    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-dshm\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.378206 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.378193    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gstpw\" (UniqueName: \"kubernetes.io/projected/9d5c4201-96c1-40b9-be00-774ccc27e376-kube-api-access-gstpw\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.479953 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.479916    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-model-cache\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.480169 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.480024    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-tmp-dir\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.480169 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.480053    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-home\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.480169 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.480079    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5c4201-96c1-40b9-be00-774ccc27e376-tls-certs\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.480169 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.480160    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-dshm\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.480389 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.480322    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gstpw\" (UniqueName: \"kubernetes.io/projected/9d5c4201-96c1-40b9-be00-774ccc27e376-kube-api-access-gstpw\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.480460 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.480439    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-model-cache\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.480600 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.480572    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-tmp-dir\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.480722 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.480601    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-home\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.482585 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.482561    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-dshm\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.482843 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.482828    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5c4201-96c1-40b9-be00-774ccc27e376-tls-certs\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.534697 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.534615    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gstpw\" (UniqueName: \"kubernetes.io/projected/9d5c4201-96c1-40b9-be00-774ccc27e376-kube-api-access-gstpw\") pod \"distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.567302 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.567268    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:32.932661 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.932631    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"]
Nov 01 12:40:32.933210 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:40:32.933185    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d5c4201_96c1_40b9_be00_774ccc27e376.slice/crio-8f7eb31f493fef65b602a4c5d0008e3d3ccb6b0eba674eb9fbdd172d7ec98a03 WatchSource:0}: Error finding container 8f7eb31f493fef65b602a4c5d0008e3d3ccb6b0eba674eb9fbdd172d7ec98a03: Status 404 returned error can't find the container with id 8f7eb31f493fef65b602a4c5d0008e3d3ccb6b0eba674eb9fbdd172d7ec98a03
Nov 01 12:40:32.954372 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:32.954344    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" event={"ID":"9d5c4201-96c1-40b9-be00-774ccc27e376","Type":"ContainerStarted","Data":"8f7eb31f493fef65b602a4c5d0008e3d3ccb6b0eba674eb9fbdd172d7ec98a03"}
Nov 01 12:40:33.961855 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:33.961819    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" event={"ID":"9d5c4201-96c1-40b9-be00-774ccc27e376","Type":"ContainerStarted","Data":"52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0"}
Nov 01 12:40:34.071298 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:34.071243    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podStartSLOduration=2.071229251 podStartE2EDuration="2.071229251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:34.069208074 +0000 UTC m=+1093.987003751" watchObservedRunningTime="2026-11-01 12:40:34.071229251 +0000 UTC m=+1093.989024916"
Nov 01 12:40:34.568202 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:34.568174    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:41.502426 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.502391    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"]
Nov 01 12:40:41.506700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.506673    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.593853 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.593819    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-dshm\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.594039 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.593879    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv5fz\" (UniqueName: \"kubernetes.io/projected/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-kube-api-access-fv5fz\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.594090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.594054    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-model-cache\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.594128 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.594114    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tls-certs\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.594315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.594300    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-home\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.594352 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.594326    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tmp-dir\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.604668 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.604640    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-06fd6b\"/\"distinct-w4-06fd6b-kserve-self-signed-certs\""
Nov 01 12:40:41.607742 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.607721    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"]
Nov 01 12:40:41.695743 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.695710    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tls-certs\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.695921 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.695896    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-home\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.695986 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.695925    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tmp-dir\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.696053 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.696029    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-dshm\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.696103 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.696094    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv5fz\" (UniqueName: \"kubernetes.io/projected/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-kube-api-access-fv5fz\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.696174 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.696157    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-model-cache\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.696362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.696332    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-home\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.696532 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.696509    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-model-cache\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.696688 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.696669    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tmp-dir\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.698298 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.698278    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-dshm\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.698776 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.698756    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tls-certs\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.750287 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.750262    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv5fz\" (UniqueName: \"kubernetes.io/projected/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-kube-api-access-fv5fz\") pod \"distinct-w4-06fd6b-kserve-7f7459846f-xzml4\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") " pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:41.819181 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:41.819112    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:42.213158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:42.213132    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"]
Nov 01 12:40:42.214180 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:40:42.214157    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce5ce330_ce56_42cb_8e11_38e3a1d0971f.slice/crio-fd6d41f9fe928e22406743533b03cd50067e84911ef7605942b6e9e2082cbd0f WatchSource:0}: Error finding container fd6d41f9fe928e22406743533b03cd50067e84911ef7605942b6e9e2082cbd0f: Status 404 returned error can't find the container with id fd6d41f9fe928e22406743533b03cd50067e84911ef7605942b6e9e2082cbd0f
Nov 01 12:40:42.567869 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:42.567787    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:42.580009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:42.579983    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:43.021786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:43.021750    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" event={"ID":"ce5ce330-ce56-42cb-8e11-38e3a1d0971f","Type":"ContainerStarted","Data":"6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88"}
Nov 01 12:40:43.021786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:43.021798    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" event={"ID":"ce5ce330-ce56-42cb-8e11-38e3a1d0971f","Type":"ContainerStarted","Data":"fd6d41f9fe928e22406743533b03cd50067e84911ef7605942b6e9e2082cbd0f"}
Nov 01 12:40:43.037215 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:43.037185    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:40:43.132341 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:43.132293    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podStartSLOduration=2.132278266 podStartE2EDuration="2.132278266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:40:43.129752403 +0000 UTC m=+1103.047548068" watchObservedRunningTime="2026-11-01 12:40:43.132278266 +0000 UTC m=+1103.050073931"
Nov 01 12:40:43.820162 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:43.820130    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:49.115278 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:40:49.115240    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:40:51.820138 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:51.820112    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:51.834184 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:51.834161    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:52.097119 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:52.097042    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:40:59.981308 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:40:59.981267    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"]
Nov 01 12:41:00.007297 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.007266    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.198001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.174795    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zhf6l\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-kube-api-access-zhf6l\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.229998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.198325    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.231808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.231724    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.299142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.299103    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.307065 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.299524    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.328600 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.328567    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.344121 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.344089    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.344471 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.344446    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.344688 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.344670    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.447569 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.447530    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"]
Nov 01 12:41:00.449473 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.449445    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.449698 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.449673    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.449915 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.449895    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.450183 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.450164    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.450383 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.450365    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.450555 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.450539    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.450682 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.450667    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.450819 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.450802    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.450960 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.450944    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zhf6l\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-kube-api-access-zhf6l\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.451895 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.451872    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.452748 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.452726    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.453402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.453364    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.453891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.453872    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.454812 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.454621    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.529043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.525557    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.530909 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.530881    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.545464 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.545420    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.553195 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.553164    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zhf6l\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-kube-api-access-zhf6l\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:00.673192 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:00.673160    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:01.139517 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:41:01.139487    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8cbbd8_ddb5_4def_a36b_e4dac5aca08a.slice/crio-425ec346dac49c40a6a13886615f36b65a309465ae9bbbb9d94a4481fc8b1799 WatchSource:0}: Error finding container 425ec346dac49c40a6a13886615f36b65a309465ae9bbbb9d94a4481fc8b1799: Status 404 returned error can't find the container with id 425ec346dac49c40a6a13886615f36b65a309465ae9bbbb9d94a4481fc8b1799
Nov 01 12:41:01.140382 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:01.140355    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"]
Nov 01 12:41:01.142061 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:01.142030    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:01.142166 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:01.142089    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:01.142166 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:01.142117    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:01.467213 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:01.467120    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" event={"ID":"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a","Type":"ContainerStarted","Data":"162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c"}
Nov 01 12:41:01.467213 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:01.467187    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" event={"ID":"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a","Type":"ContainerStarted","Data":"425ec346dac49c40a6a13886615f36b65a309465ae9bbbb9d94a4481fc8b1799"}
Nov 01 12:41:01.674337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:01.674301    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:02.676922 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:02.676886    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 12:41:02.679405 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:02.679239    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 12:41:03.677477 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:03.677437    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 12:41:03.678842 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:03.678806    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 12:41:04.686818 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:04.674305    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 12:41:04.686818 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:04.674384    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 12:41:05.678002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:05.676363    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 12:41:05.678002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:05.676425    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 12:41:06.674332 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:06.674298    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 12:41:06.674740 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:06.674353    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 12:41:07.674740 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:07.674704    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Nov 01 12:41:07.675152 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:07.674773    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Nov 01 12:41:08.678246 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:08.678211    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:08.678698 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:08.678659    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:08.679428 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:08.679411    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:41:08.889459 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:08.889404    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" podStartSLOduration=9.889389256 podStartE2EDuration="9.889389256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:40:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:01.602355997 +0000 UTC m=+1121.520151667" watchObservedRunningTime="2026-11-01 12:41:08.889389256 +0000 UTC m=+1128.807184921"
Nov 01 12:41:15.040006 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.039960    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"]
Nov 01 12:41:15.043877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.043858    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.106174 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.106142    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.106174 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.106175    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.106398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.106214    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.106398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.106373    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.106482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.106443    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.106538 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.106485    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.106538 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.106510    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.106649 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.106630    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-678ks\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-kube-api-access-678ks\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.106790 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.106774    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.143768 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.143738    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"]
Nov 01 12:41:15.147944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.147924    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"]
Nov 01 12:41:15.148091 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.148021    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.195270 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.195229    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"]
Nov 01 12:41:15.208408 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208374    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.208570 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208415    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.208570 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208448    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.208570 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208495    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.208570 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208533    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6hv7\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-kube-api-access-f6hv7\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.208808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208618    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.208808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208690    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.208808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208722    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.208808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208762    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.208808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208799    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.208808 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208802    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.209182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208843    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.209182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208901    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.209182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208934    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.209182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.208996    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-678ks\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-kube-api-access-678ks\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.209182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.209048    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.209182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.209170    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.209487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.209190    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.209487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.209338    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.209487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.209354    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.209487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.209431    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.209487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.209471    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.209487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.209469    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.211552 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.211524    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.212200 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.212176    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.262510 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.262476    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.262510 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.262489    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-678ks\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-kube-api-access-678ks\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.310584 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.310511    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6hv7\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-kube-api-access-f6hv7\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.310731 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.310616    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.310731 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.310664    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.310858 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.310807    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.310929 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.310855    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.311005 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.310947    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.311142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.311121    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.311233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.311193    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.311290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.311232    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.311346 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.311280    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-data\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.311400 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.311343    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-certs\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.311400 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.311287    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.311505 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.311478    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istiod-ca-cert\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.311671 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.311649    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-credential-socket\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.313121 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.313101    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-envoy\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.313447 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.313431    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-podinfo\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.364442 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.364419    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:15.364768 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.364745    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-token\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.365273 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.365255    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6hv7\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-kube-api-access-f6hv7\") pod \"e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") " pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.465680 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.465362    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:15.800113 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.794368    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"]
Nov 01 12:41:15.833879 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:41:15.833844    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d0c4249_1e7d_4550_b383_8b3e41ea491c.slice/crio-05270db1c900bf18102d89ae947aff7345df92cf74f31e00b2fda09cb9b23661 WatchSource:0}: Error finding container 05270db1c900bf18102d89ae947aff7345df92cf74f31e00b2fda09cb9b23661: Status 404 returned error can't find the container with id 05270db1c900bf18102d89ae947aff7345df92cf74f31e00b2fda09cb9b23661
Nov 01 12:41:15.836419 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.836384    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:15.836532 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.836465    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:15.836532 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.836513    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:15.844998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.844960    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"]
Nov 01 12:41:15.879846 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:41:15.879822    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5ab7a1_0067_4280_9028_d66c40cb9cd5.slice/crio-8819adc723f556c33dc8dc90e5361f268b6bf2c0ef41104470c70219beb32930 WatchSource:0}: Error finding container 8819adc723f556c33dc8dc90e5361f268b6bf2c0ef41104470c70219beb32930: Status 404 returned error can't find the container with id 8819adc723f556c33dc8dc90e5361f268b6bf2c0ef41104470c70219beb32930
Nov 01 12:41:15.882382 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.882355    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:15.882458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.882419    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:15.882458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:15.882448    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:16.597855 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:16.597819    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" event={"ID":"2d5ab7a1-0067-4280-9028-d66c40cb9cd5","Type":"ContainerStarted","Data":"c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5"}
Nov 01 12:41:16.598259 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:16.597869    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" event={"ID":"2d5ab7a1-0067-4280-9028-d66c40cb9cd5","Type":"ContainerStarted","Data":"8819adc723f556c33dc8dc90e5361f268b6bf2c0ef41104470c70219beb32930"}
Nov 01 12:41:16.599665 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:16.599642    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" event={"ID":"4d0c4249-1e7d-4550-b383-8b3e41ea491c","Type":"ContainerStarted","Data":"bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3"}
Nov 01 12:41:16.599784 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:16.599682    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" event={"ID":"4d0c4249-1e7d-4550-b383-8b3e41ea491c","Type":"ContainerStarted","Data":"05270db1c900bf18102d89ae947aff7345df92cf74f31e00b2fda09cb9b23661"}
Nov 01 12:41:16.726994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:16.726574    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" podStartSLOduration=1.7265543540000001 podStartE2EDuration="1.726554354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:16.723871703 +0000 UTC m=+1136.641667369" watchObservedRunningTime="2026-11-01 12:41:16.726554354 +0000 UTC m=+1136.644350020"
Nov 01 12:41:17.364765 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:17.364726    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:17.366153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:17.366127    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 12:41:17.366290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:17.366176    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 12:41:17.466341 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:17.466299    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:17.468009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:17.467960    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 12:41:17.468122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:17.468057    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 12:41:18.365698 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:18.365668    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 12:41:18.366261 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:18.366226    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 12:41:18.466471 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:18.466434    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 12:41:18.466644 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:18.466506    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 12:41:19.167474 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:41:19.167442    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:41:19.365530 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:19.365495    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 12:41:19.365705 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:19.365553    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 12:41:19.466377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:19.466292    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 12:41:19.466377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:19.466351    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 12:41:20.365313 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:20.365279    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 12:41:20.365487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:20.365330    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 12:41:20.466487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:20.466454    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 12:41:20.466874 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:20.466510    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 12:41:21.365246 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:21.365213    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 12:41:21.365433 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:21.365285    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 12:41:21.466113 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:21.466077    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 12:41:21.466285 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:21.466129    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 12:41:22.365327 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:22.365287    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body=
Nov 01 12:41:22.365728 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:22.365352    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused"
Nov 01 12:41:22.466307 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:22.466272    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body=
Nov 01 12:41:22.466481 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:22.466338    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused"
Nov 01 12:41:23.369354 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:23.369320    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:23.369756 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:23.369701    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:23.370351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:23.370329    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:41:23.470251 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:23.470221    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:23.470512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:23.470494    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:23.471153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:23.471134    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 12:41:23.575579 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:23.575529    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podStartSLOduration=9.575515664 podStartE2EDuration="9.575515664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:16.849351744 +0000 UTC m=+1136.767147411" watchObservedRunningTime="2026-11-01 12:41:23.575515664 +0000 UTC m=+1143.493311330"
Nov 01 12:41:45.296850 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.296814    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"]
Nov 01 12:41:45.301015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.300996    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.350797 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.350768    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-9514b0-openshift-default-dockercfg-4cdz5\""
Nov 01 12:41:45.394370 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.394333    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.394557 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.394532    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.394613 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.394581    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.394717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.394697    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.394781 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.394751    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453656f7-ef59-46a5-925f-37c8190d6b0f-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.394870 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.394856    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.394913 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.394900    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.394997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.394960    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.395056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.395040    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhfvc\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-kube-api-access-bhfvc\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.402021 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.401996    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"]
Nov 01 12:41:45.496261 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496229    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496431 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496271    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496431 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496303    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453656f7-ef59-46a5-925f-37c8190d6b0f-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496431 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496367    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496431 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496405    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496431 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496423    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496689 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496464    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhfvc\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-kube-api-access-bhfvc\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496689 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496524    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496689 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496639    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496857 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496726    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496857 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496745    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.496988 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.496860    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.497153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.497129    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.497220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.497186    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453656f7-ef59-46a5-925f-37c8190d6b0f-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.498773 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.498752    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.498983 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.498951    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.550762 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.550695    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.551010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.550986    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhfvc\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-kube-api-access-bhfvc\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:45.615050 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:45.615014    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:46.031020 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:46.030990    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"]
Nov 01 12:41:46.065848 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:41:46.065816    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod453656f7_ef59_46a5_925f_37c8190d6b0f.slice/crio-747954627de71c46a9200b57dac54070ec1c0c88bfa613d55ba6326f1e5df560 WatchSource:0}: Error finding container 747954627de71c46a9200b57dac54070ec1c0c88bfa613d55ba6326f1e5df560: Status 404 returned error can't find the container with id 747954627de71c46a9200b57dac54070ec1c0c88bfa613d55ba6326f1e5df560
Nov 01 12:41:46.068220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:46.068191    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:46.068298 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:46.068256    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:46.068298 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:46.068285    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:41:46.840316 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:46.840277    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" event={"ID":"453656f7-ef59-46a5-925f-37c8190d6b0f","Type":"ContainerStarted","Data":"0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf"}
Nov 01 12:41:46.840813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:46.840346    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" event={"ID":"453656f7-ef59-46a5-925f-37c8190d6b0f","Type":"ContainerStarted","Data":"747954627de71c46a9200b57dac54070ec1c0c88bfa613d55ba6326f1e5df560"}
Nov 01 12:41:46.950751 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:46.950695    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podStartSLOduration=1.950670195 podStartE2EDuration="1.950670195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:41:46.949811988 +0000 UTC m=+1166.867607654" watchObservedRunningTime="2026-11-01 12:41:46.950670195 +0000 UTC m=+1166.868465862"
Nov 01 12:41:47.620375 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:47.620338    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:47.625098 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:47.625067    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 12:41:47.625351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:47.625313    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 12:41:48.624582 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:48.624548    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 12:41:48.628088 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:48.628046    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 12:41:49.136630 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:41:49.136590    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:41:49.626992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:49.622393    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 12:41:49.626992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:49.622454    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 12:41:50.622996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:50.616451    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 12:41:50.622996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:50.616516    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 12:41:51.626996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:51.620259    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 12:41:51.626996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:51.620316    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 12:41:52.616220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:52.616188    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 01 12:41:52.616444 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:52.616243    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 01 12:41:53.620637 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:53.620602    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:53.621055 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:53.620873    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:41:53.621699 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:41:53.621683    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:42:06.578356 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.578321    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"]
Nov 01 12:42:06.583750 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.583726    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.635075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.635038    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"]
Nov 01 12:42:06.678875 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.678847    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-5b1bd4\"/\"distinct-2-w1-5b1bd4-kserve-self-signed-certs\""
Nov 01 12:42:06.689344 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.689305    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70639c6-8601-4a9c-866c-d194156300b6-tls-certs\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.689543 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.689433    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-dshm\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.689622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.689545    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69v5x\" (UniqueName: \"kubernetes.io/projected/c70639c6-8601-4a9c-866c-d194156300b6-kube-api-access-69v5x\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.689622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.689617    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-model-cache\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.689717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.689641    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-tmp-dir\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.689771 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.689756    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-home\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.791369 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.791335    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-home\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.791543 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.791415    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70639c6-8601-4a9c-866c-d194156300b6-tls-certs\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.791543 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.791467    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-dshm\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.791543 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.791527    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69v5x\" (UniqueName: \"kubernetes.io/projected/c70639c6-8601-4a9c-866c-d194156300b6-kube-api-access-69v5x\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.791679 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.791568    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-model-cache\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.791679 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.791586    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-tmp-dir\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.791843 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.791728    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-home\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.791947 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.791926    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-model-cache\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.792090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.792071    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-tmp-dir\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.793799 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.793777    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-dshm\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.794016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.794001    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70639c6-8601-4a9c-866c-d194156300b6-tls-certs\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.845125 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.845062    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69v5x\" (UniqueName: \"kubernetes.io/projected/c70639c6-8601-4a9c-866c-d194156300b6-kube-api-access-69v5x\") pod \"distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") " pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:06.899446 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:06.899363    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:07.289539 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:07.289510    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"]
Nov 01 12:42:07.290438 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:42:07.290415    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc70639c6_8601_4a9c_866c_d194156300b6.slice/crio-cc20a95e6824795288fc03ecc7720509ec5205b634b779caa46f6301e0a63654 WatchSource:0}: Error finding container cc20a95e6824795288fc03ecc7720509ec5205b634b779caa46f6301e0a63654: Status 404 returned error can't find the container with id cc20a95e6824795288fc03ecc7720509ec5205b634b779caa46f6301e0a63654
Nov 01 12:42:07.997364 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:07.997329    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" event={"ID":"c70639c6-8601-4a9c-866c-d194156300b6","Type":"ContainerStarted","Data":"863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5"}
Nov 01 12:42:07.997364 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:07.997370    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" event={"ID":"c70639c6-8601-4a9c-866c-d194156300b6","Type":"ContainerStarted","Data":"cc20a95e6824795288fc03ecc7720509ec5205b634b779caa46f6301e0a63654"}
Nov 01 12:42:08.106467 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:08.106409    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podStartSLOduration=2.106395045 podStartE2EDuration="2.106395045s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:08.104898305 +0000 UTC m=+1188.022693972" watchObservedRunningTime="2026-11-01 12:42:08.106395045 +0000 UTC m=+1188.024190736"
Nov 01 12:42:08.900333 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:08.900306    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:15.384177 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.384129    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"]
Nov 01 12:42:15.388527 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.388506    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.488678 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.488646    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"]
Nov 01 12:42:15.516848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.516810    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.517021 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.516938    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.517021 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.516994    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.517119 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.517052    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z25rj\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-kube-api-access-z25rj\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.517119 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.517106    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c942a0-8f71-44e6-a432-a3f8a69ec808-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.517231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.517201    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.517279 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.517242    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.517347 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.517331    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.517443 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.517429    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619120 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619081    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619120 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619120    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619149    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z25rj\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-kube-api-access-z25rj\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619213    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c942a0-8f71-44e6-a432-a3f8a69ec808-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619352    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619396    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619431    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619672 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619615    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619672 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619629    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619784 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619714    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-credential-socket\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619784 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619740    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.619916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.619823    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-data\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.620042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.620023    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c942a0-8f71-44e6-a432-a3f8a69ec808-istiod-ca-cert\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.620109 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.620052    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-certs\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.621513 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.621488    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-envoy\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.621759 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.621740    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-podinfo\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.672919 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.672854    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-token\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.672919 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.672875    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z25rj\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-kube-api-access-z25rj\") pod \"e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") " pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:15.704440 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:15.704418    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:16.119793 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:16.119769    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"]
Nov 01 12:42:16.120691 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:42:16.120657    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89c942a0_8f71_44e6_a432_a3f8a69ec808.slice/crio-43fc47d105b24f08b1b22d42d727c9a6aad8def0d812a6057465b1e002f1dae2 WatchSource:0}: Error finding container 43fc47d105b24f08b1b22d42d727c9a6aad8def0d812a6057465b1e002f1dae2: Status 404 returned error can't find the container with id 43fc47d105b24f08b1b22d42d727c9a6aad8def0d812a6057465b1e002f1dae2
Nov 01 12:42:16.123896 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:16.123863    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:42:16.123998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:16.123928    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:42:16.123998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:16.123958    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:42:16.900204 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:16.900170    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:16.940060 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:16.940031    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:17.053737 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:17.053701    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" event={"ID":"89c942a0-8f71-44e6-a432-a3f8a69ec808","Type":"ContainerStarted","Data":"850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970"}
Nov 01 12:42:17.053902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:17.053756    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" event={"ID":"89c942a0-8f71-44e6-a432-a3f8a69ec808","Type":"ContainerStarted","Data":"43fc47d105b24f08b1b22d42d727c9a6aad8def0d812a6057465b1e002f1dae2"}
Nov 01 12:42:17.065106 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:17.065081    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:42:17.185710 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:17.185616    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" podStartSLOduration=2.185602203 podStartE2EDuration="2.185602203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:17.183757073 +0000 UTC m=+1197.101552750" watchObservedRunningTime="2026-11-01 12:42:17.185602203 +0000 UTC m=+1197.103397868"
Nov 01 12:42:17.715114 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:17.709258    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:17.719404 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:17.719076    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 12:42:17.719404 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:17.719131    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 12:42:18.724385 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:18.724349    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 12:42:18.725094 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:18.725063    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 12:42:19.139730 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:42:19.139690    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:42:19.709411 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:19.709372    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 12:42:19.709727 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:19.709696    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 12:42:20.317997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:20.302046    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:42:20.334443 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:20.333127    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:42:20.718344 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:20.718304    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 12:42:20.718637 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:20.718607    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 12:42:21.706400 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:21.706360    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 12:42:21.707057 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:21.707019    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 12:42:22.705069 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:22.705022    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body=
Nov 01 12:42:22.705250 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:22.705093    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused"
Nov 01 12:42:23.709043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:23.709012    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:23.709451 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:23.709347    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:23.710149 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:23.710129    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:42:45.734657 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.734573    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"]
Nov 01 12:42:45.738864 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.738845    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.783700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.783664    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.783842 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.783709    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.783892 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.783878    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.783947 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.783935    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8r94\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-kube-api-access-h8r94\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.783995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.783978    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.784064 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.784051    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af74efd4-549a-4129-9c9e-01e27494f9e4-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.784180 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.784159    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.784217 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.784201    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.784293 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.784280    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.885589 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.885555    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af74efd4-549a-4129-9c9e-01e27494f9e4-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.885751 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.885615    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.885751 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.885662    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.885751 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.885715    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.885853 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.885802    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.885853 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.885834    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.885940 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.885924    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.886060 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.886039    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8r94\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-kube-api-access-h8r94\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.886135 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.886057    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-credential-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.886135 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.886087    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.886313 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.886142    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-certs\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.886313 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.886163    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-data\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.886438 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.886414    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af74efd4-549a-4129-9c9e-01e27494f9e4-istiod-ca-cert\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.886499 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.886463    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-socket\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.888144 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.888118    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-envoy\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.889740 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.889718    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-podinfo\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.930568 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.930538    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"]
Nov 01 12:42:45.940237 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.940208    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-token\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:45.940615 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:45.940596    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8r94\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-kube-api-access-h8r94\") pod \"e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") " pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:46.053269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:46.053183    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:46.477185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:46.477158    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"]
Nov 01 12:42:46.477612 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:42:46.477584    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf74efd4_549a_4129_9c9e_01e27494f9e4.slice/crio-649a30545f33cd412f0f79e692ef6bfad8823fb2583265f7a3bc5ed6f192c754 WatchSource:0}: Error finding container 649a30545f33cd412f0f79e692ef6bfad8823fb2583265f7a3bc5ed6f192c754: Status 404 returned error can't find the container with id 649a30545f33cd412f0f79e692ef6bfad8823fb2583265f7a3bc5ed6f192c754
Nov 01 12:42:46.480176 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:46.480143    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:42:46.480261 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:46.480212    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:42:46.480261 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:46.480240    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:42:47.288053 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:47.288008    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" event={"ID":"af74efd4-549a-4129-9c9e-01e27494f9e4","Type":"ContainerStarted","Data":"aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962"}
Nov 01 12:42:47.288565 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:47.288082    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" event={"ID":"af74efd4-549a-4129-9c9e-01e27494f9e4","Type":"ContainerStarted","Data":"649a30545f33cd412f0f79e692ef6bfad8823fb2583265f7a3bc5ed6f192c754"}
Nov 01 12:42:47.449555 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:47.449487    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" podStartSLOduration=2.449469869 podStartE2EDuration="2.449469869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:42:47.446376806 +0000 UTC m=+1227.364172473" watchObservedRunningTime="2026-11-01 12:42:47.449469869 +0000 UTC m=+1227.367265639"
Nov 01 12:42:48.053366 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:48.053332    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:48.055212 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:48.055178    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 12:42:48.055341 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:48.055249    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 12:42:49.056081 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:49.056042    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 12:42:49.056472 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:49.056108    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 12:42:49.143172 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:42:49.143141    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:42:50.053931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:50.053889    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 12:42:50.054209 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:50.053955    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 12:42:51.053605 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:51.053575    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 12:42:51.054043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:51.053631    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 12:42:52.054134 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:52.054104    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 12:42:52.054544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:52.054162    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 12:42:53.054544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:53.054491    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 01 12:42:53.055004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:53.054557    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 01 12:42:54.058912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:54.058883    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:54.059351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:54.059169    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:42:54.059855 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:42:54.059837    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 12:43:02.378302 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.378266    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"]
Nov 01 12:43:02.382719 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.382702    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.479839 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.479806    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-94c94b\"/\"distinct-2-w0-94c94b-kserve-self-signed-certs\""
Nov 01 12:43:02.483192 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.483166    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"]
Nov 01 12:43:02.487759 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.487726    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-tmp-dir\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.487850 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.487780    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-home\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.487998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.487962    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lcr9\" (UniqueName: \"kubernetes.io/projected/3f047c32-f259-4e7e-b98b-3772fe66aa55-kube-api-access-9lcr9\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.488075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.488017    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f047c32-f259-4e7e-b98b-3772fe66aa55-tls-certs\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.488182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.488169    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-model-cache\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.488282 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.488269    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-dshm\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.589580 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.589541    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-model-cache\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.589785 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.589649    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-dshm\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.589785 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.589751    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-tmp-dir\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.589902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.589796    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-home\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.589959 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.589942    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lcr9\" (UniqueName: \"kubernetes.io/projected/3f047c32-f259-4e7e-b98b-3772fe66aa55-kube-api-access-9lcr9\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.589959 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.589946    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-model-cache\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.590111 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.589989    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f047c32-f259-4e7e-b98b-3772fe66aa55-tls-certs\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.590111 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.590099    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-tmp-dir\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.590199 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.590176    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-home\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.591938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.591917    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-dshm\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.592436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.592416    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f047c32-f259-4e7e-b98b-3772fe66aa55-tls-certs\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.645743 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.645670    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lcr9\" (UniqueName: \"kubernetes.io/projected/3f047c32-f259-4e7e-b98b-3772fe66aa55-kube-api-access-9lcr9\") pod \"distinct-2-w0-94c94b-kserve-fd9756449-88csk\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") " pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:02.695510 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:02.695474    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:03.064385 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:03.064354    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"]
Nov 01 12:43:03.064513 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:43:03.064361    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f047c32_f259_4e7e_b98b_3772fe66aa55.slice/crio-6068e72ae1a243ee36338f3cfddc99680b72d3bc8e03110f63ed70e277da0644 WatchSource:0}: Error finding container 6068e72ae1a243ee36338f3cfddc99680b72d3bc8e03110f63ed70e277da0644: Status 404 returned error can't find the container with id 6068e72ae1a243ee36338f3cfddc99680b72d3bc8e03110f63ed70e277da0644
Nov 01 12:43:03.450516 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:03.450479    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" event={"ID":"3f047c32-f259-4e7e-b98b-3772fe66aa55","Type":"ContainerStarted","Data":"557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136"}
Nov 01 12:43:03.450890 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:03.450527    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" event={"ID":"3f047c32-f259-4e7e-b98b-3772fe66aa55","Type":"ContainerStarted","Data":"6068e72ae1a243ee36338f3cfddc99680b72d3bc8e03110f63ed70e277da0644"}
Nov 01 12:43:03.561277 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:03.561225    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podStartSLOduration=1.561211654 podStartE2EDuration="1.561211654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:03.558729997 +0000 UTC m=+1243.476525664" watchObservedRunningTime="2026-11-01 12:43:03.561211654 +0000 UTC m=+1243.479007319"
Nov 01 12:43:03.697531 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:03.697485    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:07.311119 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.311084    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"]
Nov 01 12:43:07.316109 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.316087    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.413695 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.413664    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"openshift-service-ca.crt\""
Nov 01 12:43:07.413695 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.413665    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"default-dockercfg-wj74h\""
Nov 01 12:43:07.413929 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.413707    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"distinct-2-w3-ecd640-kserve-self-signed-certs\""
Nov 01 12:43:07.413929 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.413772    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-ecd640\"/\"kube-root-ca.crt\""
Nov 01 12:43:07.419450 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.419425    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"]
Nov 01 12:43:07.460627 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.460591    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tmp-dir\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.460818 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.460718    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-home\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.460818 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.460772    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfnnr\" (UniqueName: \"kubernetes.io/projected/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-kube-api-access-xfnnr\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.462850 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.462237    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-dshm\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.462850 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.462293    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-model-cache\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.462850 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.462355    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tls-certs\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.564443 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.564329    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-dshm\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.564443 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.564394    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-model-cache\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.564695 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.564459    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tls-certs\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.564695 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.564559    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tmp-dir\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.564695 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.564603    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-home\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.564695 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.564637    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfnnr\" (UniqueName: \"kubernetes.io/projected/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-kube-api-access-xfnnr\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.565440 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.565414    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tmp-dir\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.565670 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.565600    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-home\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.566103 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.566080    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-model-cache\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.568711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.568689    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-dshm\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.571253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.571229    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tls-certs\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.620239 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.620205    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfnnr\" (UniqueName: \"kubernetes.io/projected/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-kube-api-access-xfnnr\") pod \"distinct-2-w3-ecd640-kserve-84d797b4c-295q6\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") " pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.629723 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.629695    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:07.997997 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:43:07.997954    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd60a0271_a7d9_4a8c_8bd3_34ad08fb0012.slice/crio-d9f6cfe1d3ea7c02406aed62bff9feea5e99605874c3963928bd9414454ece14 WatchSource:0}: Error finding container d9f6cfe1d3ea7c02406aed62bff9feea5e99605874c3963928bd9414454ece14: Status 404 returned error can't find the container with id d9f6cfe1d3ea7c02406aed62bff9feea5e99605874c3963928bd9414454ece14
Nov 01 12:43:07.998166 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:07.998049    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"]
Nov 01 12:43:08.499383 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:08.499347    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" event={"ID":"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012","Type":"ContainerStarted","Data":"b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25"}
Nov 01 12:43:08.499383 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:08.499388    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" event={"ID":"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012","Type":"ContainerStarted","Data":"d9f6cfe1d3ea7c02406aed62bff9feea5e99605874c3963928bd9414454ece14"}
Nov 01 12:43:08.630493 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:08.630461    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:12.696697 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:12.696657    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:12.709353 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:12.709323    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:12.820724 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:12.820672    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podStartSLOduration=5.820657869 podStartE2EDuration="5.820657869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:08.610437606 +0000 UTC m=+1248.528233272" watchObservedRunningTime="2026-11-01 12:43:12.820657869 +0000 UTC m=+1252.738453535"
Nov 01 12:43:13.541924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:13.541896    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:43:16.827255 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:16.825043    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.059004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.058051    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:43:17.095047 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.094224    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.095224 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.095112    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.119996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.116132    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 12:43:17.175997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.174146    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.211994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.208043    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 12:43:17.379143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.379083    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.545996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.545084    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.616190 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.616143    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.637194 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.637108    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:17.684517 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.684422    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.834992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.834164    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:17.834992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.834964    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:43:17.916199 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:17.915058    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:18.160008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.115853    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:43:18.160008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.116572    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:43:18.160008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.143072    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:18.160008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.143212    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 12:43:18.160008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.144551    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:43:18.160008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.158076    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:18.160008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.158495    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:43:18.213051 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.175326    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:18.388989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.238512    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:18.389198 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.386661    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:43:18.432263 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.432218    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:43:18.432588 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.432569    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:43:18.432775 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.432752    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:43:18.432947 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.432924    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:43:18.446034 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.438626    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:18.449854 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.449836    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:43:18.467947 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.467884    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:43:18.479701 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.476609    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:43:18.497684 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.495705    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:43:18.497684 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.495856    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:43:18.505688 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.503934    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:43:18.505688 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:18.505407    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:43:19.115700 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:43:19.115665    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:43:33.197510 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.197477    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"]
Nov 01 12:43:33.203040 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.203021    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.297021 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.296990    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"]
Nov 01 12:43:33.309607 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.309557    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7qdn\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-kube-api-access-w7qdn\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.309777 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.309659    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.309777 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.309688    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.309872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.309808    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faf786a-e922-45d7-a7b1-922e2929650b-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.309925 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.309905    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.310113 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.310088    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.310225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.310199    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.310266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.310251    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faf786a-e922-45d7-a7b1-922e2929650b-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.310373 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.310358    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.411531 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.411501    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.411531 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.411538    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.411761 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.411659    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faf786a-e922-45d7-a7b1-922e2929650b-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.411761 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.411753    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.411846 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.411803    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.411846 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.411836    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.411923 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.411875    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faf786a-e922-45d7-a7b1-922e2929650b-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.411982 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.411937    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.412038 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.411960    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.412079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.412059    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.412135 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.412097    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7qdn\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-kube-api-access-w7qdn\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.412290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.412244    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.413324 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.413297    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faf786a-e922-45d7-a7b1-922e2929650b-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.413781 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.413756    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.414639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.414615    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faf786a-e922-45d7-a7b1-922e2929650b-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.414744 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.414723    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.469361 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.469277    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7qdn\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-kube-api-access-w7qdn\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.469541 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.469439    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.517907 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.517874    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:33.937307 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.937285    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"]
Nov 01 12:43:33.937867 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:43:33.937842    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3faf786a_e922_45d7_a7b1_922e2929650b.slice/crio-f8bfad03c985d03ba81783c98e3f4d0e27a9a6955d955c4823ba6a8e870d8442 WatchSource:0}: Error finding container f8bfad03c985d03ba81783c98e3f4d0e27a9a6955d955c4823ba6a8e870d8442: Status 404 returned error can't find the container with id f8bfad03c985d03ba81783c98e3f4d0e27a9a6955d955c4823ba6a8e870d8442
Nov 01 12:43:33.940108 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.940080    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:43:33.940176 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.940151    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:43:33.940208 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:33.940179    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:43:34.819306 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:34.819262    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" event={"ID":"3faf786a-e922-45d7-a7b1-922e2929650b","Type":"ContainerStarted","Data":"9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264"}
Nov 01 12:43:34.819906 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:34.819878    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" event={"ID":"3faf786a-e922-45d7-a7b1-922e2929650b","Type":"ContainerStarted","Data":"f8bfad03c985d03ba81783c98e3f4d0e27a9a6955d955c4823ba6a8e870d8442"}
Nov 01 12:43:34.991766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:34.991718    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" podStartSLOduration=1.991703812 podStartE2EDuration="1.991703812s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:34.989514948 +0000 UTC m=+1274.907310614" watchObservedRunningTime="2026-11-01 12:43:34.991703812 +0000 UTC m=+1274.909499477"
Nov 01 12:43:35.520017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:35.519984    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:35.527162 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:35.527116    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:43:35.527460 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:35.527415    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 12:43:36.524992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:36.524065    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:43:36.524992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:36.524127    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 12:43:37.524993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:37.524256    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:43:37.524993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:37.524319    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 12:43:38.536997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:38.529457    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:43:38.536997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:38.529526    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 12:43:39.526994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:39.526850    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:43:39.526994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:39.526908    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 12:43:40.518857 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:40.518817    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 01 12:43:40.519277 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:40.518892    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 01 12:43:41.315190 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.315159    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"]
Nov 01 12:43:41.319348 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.319333    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.369296 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.369267    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 01 12:43:41.369296 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.369289    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 01 12:43:41.369567 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.369296    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 01 12:43:41.369567 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.369348    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-mmr7k\""
Nov 01 12:43:41.421914 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.421881    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"]
Nov 01 12:43:41.460088 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.460056    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.460277 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.460190    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4328f10-aaee-41f8-8c0c-5fb584f00662-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.460354 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.460340    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.460413 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.460374    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.460485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.460469    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.460558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.460540    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.460616 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.460580    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmx2z\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-kube-api-access-pmx2z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.523339 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.523312    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:41.523789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.523769    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:41.524441 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.524422    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:43:41.562529 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.562495    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.562741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.562565    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.562741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.562595    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmx2z\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-kube-api-access-pmx2z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.562741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.562711    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.563016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.562766    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4328f10-aaee-41f8-8c0c-5fb584f00662-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.563016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.562869    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.563016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.562908    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.563571 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.563544    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.565188 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.565162    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.565325 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.565212    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.565325 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.565251    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4328f10-aaee-41f8-8c0c-5fb584f00662-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.565325 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.565315    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.616634 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.616601    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.616801 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.616743    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmx2z\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-kube-api-access-pmx2z\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2n9j4\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:41.632843 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:41.632815    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:42.051272 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:42.051245    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"]
Nov 01 12:43:42.051598 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:43:42.051575    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4328f10_aaee_41f8_8c0c_5fb584f00662.slice/crio-24fa87c5696143247b212280a9fe0479b8b602896ad3d59e9516e7d709973a4b WatchSource:0}: Error finding container 24fa87c5696143247b212280a9fe0479b8b602896ad3d59e9516e7d709973a4b: Status 404 returned error can't find the container with id 24fa87c5696143247b212280a9fe0479b8b602896ad3d59e9516e7d709973a4b
Nov 01 12:43:42.053537 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:42.053522    4095 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:43:42.883244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:42.883199    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4" event={"ID":"b4328f10-aaee-41f8-8c0c-5fb584f00662","Type":"ContainerStarted","Data":"24fa87c5696143247b212280a9fe0479b8b602896ad3d59e9516e7d709973a4b"}
Nov 01 12:43:46.272777 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:46.272718    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:43:46.273050 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:46.272817    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:43:46.914491 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:46.914459    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4" event={"ID":"b4328f10-aaee-41f8-8c0c-5fb584f00662","Type":"ContainerStarted","Data":"f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8"}
Nov 01 12:43:46.914716 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:46.914510    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:43:47.120060 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:47.120010    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4" podStartSLOduration=1.902197191 podStartE2EDuration="6.119993357s" totalImagesPullingTime="4.217796166s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:43:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:43:47.071541094 +0000 UTC m=+1286.989336760" watchObservedRunningTime="2026-11-01 12:43:47.119993357 +0000 UTC m=+1287.037789019"
Nov 01 12:43:47.923544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:47.923513    4095 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2n9j4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:43:47.923916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:47.923572    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4" podUID="b4328f10-aaee-41f8-8c0c-5fb584f00662" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:43:49.203366 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:43:49.203326    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:43:50.926760 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:43:50.926729    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 12:44:14.392995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.392067    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 12:44:14.464996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.464083    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.468995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.465519    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 01 12:44:14.482192 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.482131    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.484961 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.484631    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.499161 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.499112    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.512203 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.512147    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 12:44:14.512903 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.512860    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 01 12:44:14.540867 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.540803    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 01 12:44:14.616182 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.616119    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:14.685579 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:14.685480    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.392996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.392904    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 12:44:15.393794 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.393647    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:44:15.420671 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.419961    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded"
Nov 01 12:44:15.470000 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.465057    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.470000 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.465508    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:44:15.493003 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.490057    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.493003 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.492180    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:44:15.494010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.493335    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 12:44:15.494010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.493568    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.499856    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.500243    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.526080    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.526490    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.527451    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.527769    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.528519    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.528788    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.543042    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 01 12:44:15.546994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.543394    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:44:15.664053 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.626938    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 12:44:15.792002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.728689    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:44:15.793281 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.793252    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:44:15.796999 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:15.793484    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:44:16.396151 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.396108    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:16.473392 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.473349    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:16.626997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.606129    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 01 12:44:16.846961 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.646944    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:16.848021 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.647200    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 01 12:44:16.848649 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.647436    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 01 12:44:16.874994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.739293    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:16.908996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.905124    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:16.998042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.739515    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 12:44:16.998042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.942314    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:44:16.998042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:16.942528    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 12:44:17.434433 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:17.434382    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded"
Nov 01 12:44:17.458343 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:17.458102    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:17.496397 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:17.496341    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 12:44:17.557481 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:17.557453    4095 trace.go:236] Trace[721941911]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-fcfb9fcfb-brd47" (01-Nov-2026 12:44:16.020) (total time: 1537ms):
Nov 01 12:44:17.557481 ip-10-0-140-168 kubenswrapper[4095]: Trace[721941911]: [1.53716508s] [1.53716508s] END
Nov 01 12:44:17.797060 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:44:17.763551    4095 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.557s"
Nov 01 12:44:17.828409 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:17.828358    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:17.885992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:17.862054    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:17.916993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:17.893024    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.001863 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.001813    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 12:44:18.128009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.111876    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"]
Nov 01 12:44:18.128009 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.002116    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 01 12:44:18.128263 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.002609    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:44:18.128430 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.002793    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 12:44:18.157994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.002815    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:44:18.158499 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.002189    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.203996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.132066    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.203996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.137511    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.203996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.177754    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"]
Nov 01 12:44:18.294994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.264651    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-data\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.294994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.264719    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6706ea89-410e-4c88-9a21-820078f1c1d6-istiod-ca-cert\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.294994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.264826    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-credential-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.294994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.264872    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grmks\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-kube-api-access-grmks\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.294994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.264926    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-certs\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.294994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.285210    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.294994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.285269    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-podinfo\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.294994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.285452    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-envoy\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.294994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.285491    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-token\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.421211    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-ecd640-openshift-default-dockercfg-mfhth\""
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.433601    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-token\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.457213    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-data\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.457285    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6706ea89-410e-4c88-9a21-820078f1c1d6-istiod-ca-cert\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.457415    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-credential-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.457469    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grmks\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-kube-api-access-grmks\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.457515    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-certs\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.457592    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.457629    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-podinfo\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.458143 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.457788    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-envoy\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.459133 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.458593    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-credential-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.459133 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.458864    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-data\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.459907 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.459521    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6706ea89-410e-4c88-9a21-820078f1c1d6-istiod-ca-cert\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.459907 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.459755    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-certs\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.461240 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.461204    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-socket\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.466305 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.462874    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.466305 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.463608    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"]
Nov 01 12:44:18.506401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.506354    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.563074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.557523    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:18.573997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.573948    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-token\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.590179 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.589845    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grmks\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-kube-api-access-grmks\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.606717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.606627    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-envoy\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.617210 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.616655    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-podinfo\") pod \"e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") " pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.658397 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.658367    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:44:18.658625 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.658614    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:44:18.658743 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.658732    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"]
Nov 01 12:44:18.660944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.659813    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.664911 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.664879    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded"
Nov 01 12:44:18.665789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.665757    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"]
Nov 01 12:44:18.733390 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.731240    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:18.768954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.767954    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.768954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.768007    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.768954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.768048    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.768954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.768105    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.768954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.768231    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xm96m\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-kube-api-access-xm96m\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.768954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.768276    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.768954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.768371    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.768954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.768505    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.768954 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.768554    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.795122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.781497    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:44:18.795122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.781545    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:44:18.795122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.781565    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"]
Nov 01 12:44:18.795122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.781612    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:44:18.795122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.781654    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:44:18.795122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.781685    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:44:18.795122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.781714    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:44:18.795122 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.782172    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:18.829044 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.829015    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:44:18.829268 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.829095    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:44:18.829268 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.829130    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:44:18.872840 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.872813    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xm96m\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-kube-api-access-xm96m\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.873137 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873118    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.873381 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873363    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.874017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873551    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.874017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873406    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.874017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873630    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.874017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873663    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.874017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873740    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.874017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873756    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.874017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873779    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.874017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.873816    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.874493 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.874409    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.875207 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.875186    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.875378 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.875205    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.879528 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.879504    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.880725 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.880708    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.929841 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.929809    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.930010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.929853    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xm96m\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-kube-api-access-xm96m\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:18.974917 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.974885    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42hvk\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-kube-api-access-42hvk\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:18.975111 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.975056    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:18.975184 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.975141    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:18.975244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.975196    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:18.975244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.975215    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:18.975635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.975611    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:18.975801 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.975767    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:18.975916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.975819    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:18.976037 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:18.976016    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.077896 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.077812    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.077896 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.077881    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.078153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.077910    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.078153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.078128    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.078270 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.078202    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.078270 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.078231    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.078383 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.078332    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.078446 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.078394    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42hvk\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-kube-api-access-42hvk\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.078502 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.078466    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.078748 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.078712    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-certs\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.078840 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.078747    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istiod-ca-cert\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.079108 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.079084    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-credential-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.079199 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.079112    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-socket\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.079258 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.079196    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-data\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.081208 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.081183    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-envoy\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.082773 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.082743    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-podinfo\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.128618 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:44:19.128576    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:44:19.133732 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.133695    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-token\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.133873 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.133804    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42hvk\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-kube-api-access-42hvk\") pod \"e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") " pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.158046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.158020    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:19.173515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.173492    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:19.237737 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:44:19.237707    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6706ea89_410e_4c88_9a21_820078f1c1d6.slice/crio-794b6949ad64770c2e742494f3424fc48e4d362fd7a63c0c12bbe7b3034cd194 WatchSource:0}: Error finding container 794b6949ad64770c2e742494f3424fc48e4d362fd7a63c0c12bbe7b3034cd194: Status 404 returned error can't find the container with id 794b6949ad64770c2e742494f3424fc48e4d362fd7a63c0c12bbe7b3034cd194
Nov 01 12:44:19.244577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.244552    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"]
Nov 01 12:44:19.248360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.247123    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:44:19.248360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.247206    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:44:19.248360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.247250    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:44:19.478067 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.478034    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" event={"ID":"6706ea89-410e-4c88-9a21-820078f1c1d6","Type":"ContainerStarted","Data":"8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6"}
Nov 01 12:44:19.478436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.478091    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" event={"ID":"6706ea89-410e-4c88-9a21-820078f1c1d6","Type":"ContainerStarted","Data":"794b6949ad64770c2e742494f3424fc48e4d362fd7a63c0c12bbe7b3034cd194"}
Nov 01 12:44:19.552627 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.552593    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"]
Nov 01 12:44:19.556564 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:44:19.556526    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4958fe8c_32cc_41b5_a2e9_e81fbf866721.slice/crio-5e706f83139d1d70984b004a99c8fbf574abd239c29d76b5ed25a98b89fcae15 WatchSource:0}: Error finding container 5e706f83139d1d70984b004a99c8fbf574abd239c29d76b5ed25a98b89fcae15: Status 404 returned error can't find the container with id 5e706f83139d1d70984b004a99c8fbf574abd239c29d76b5ed25a98b89fcae15
Nov 01 12:44:19.560021 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.559984    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:44:19.560134 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.560070    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:44:19.560134 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.560117    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:44:19.597908 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.597882    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"]
Nov 01 12:44:19.607887 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:44:19.607852    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7809a1c2_11c0_4d8d_90e8_9510a6f3d915.slice/crio-4df176a317208a4a29a9805f28dfd9a404a89b2b298b56f70dd5bb1912b6b731 WatchSource:0}: Error finding container 4df176a317208a4a29a9805f28dfd9a404a89b2b298b56f70dd5bb1912b6b731: Status 404 returned error can't find the container with id 4df176a317208a4a29a9805f28dfd9a404a89b2b298b56f70dd5bb1912b6b731
Nov 01 12:44:19.610844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.610808    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:44:19.610932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.610899    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:44:19.610989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.610945    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:44:19.762942 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:19.759200    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:20.360436 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:20.360373    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" podStartSLOduration=4.360356748 podStartE2EDuration="4.360356748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:20.356750374 +0000 UTC m=+1320.274546040" watchObservedRunningTime="2026-11-01 12:44:20.360356748 +0000 UTC m=+1320.278152416"
Nov 01 12:44:20.486630 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:20.486590    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" event={"ID":"4958fe8c-32cc-41b5-a2e9-e81fbf866721","Type":"ContainerStarted","Data":"db36717e60686331532d72fbb54c14352d385452ca20218369a3b1cd10e17321"}
Nov 01 12:44:20.487140 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:20.486693    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" event={"ID":"4958fe8c-32cc-41b5-a2e9-e81fbf866721","Type":"ContainerStarted","Data":"5e706f83139d1d70984b004a99c8fbf574abd239c29d76b5ed25a98b89fcae15"}
Nov 01 12:44:20.488392 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:20.488358    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" event={"ID":"7809a1c2-11c0-4d8d-90e8-9510a6f3d915","Type":"ContainerStarted","Data":"48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77"}
Nov 01 12:44:20.488547 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:20.488427    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" event={"ID":"7809a1c2-11c0-4d8d-90e8-9510a6f3d915","Type":"ContainerStarted","Data":"4df176a317208a4a29a9805f28dfd9a404a89b2b298b56f70dd5bb1912b6b731"}
Nov 01 12:44:20.655096 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:20.652430    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" podStartSLOduration=4.652414656 podStartE2EDuration="4.652414656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:20.612359496 +0000 UTC m=+1320.530155163" watchObservedRunningTime="2026-11-01 12:44:20.652414656 +0000 UTC m=+1320.570210321"
Nov 01 12:44:20.754006 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:20.753832    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" podStartSLOduration=2.75381669 podStartE2EDuration="2.75381669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:44:20.75357099 +0000 UTC m=+1320.671366657" watchObservedRunningTime="2026-11-01 12:44:20.75381669 +0000 UTC m=+1320.671612359"
Nov 01 12:44:20.797993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:20.773753    4095 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 12:44:20.797993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:20.773821    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 12:44:21.166123 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:21.166089    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:21.183155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:21.176117    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 12:44:21.183155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:21.176180    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 12:44:21.183155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:21.176460    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:21.183155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:21.178036    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 12:44:21.183155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:21.178084    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 12:44:21.759786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:21.759351    4095 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 12:44:21.759786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:21.759413    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 12:44:22.169995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:22.160861    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 12:44:22.169995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:22.160935    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 12:44:22.179997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:22.175277    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 12:44:22.179997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:22.175335    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 12:44:22.791994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:22.782780    4095 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 12:44:22.791994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:22.782837    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 12:44:23.160996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:23.160679    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 12:44:23.189542 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:23.186168    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 12:44:23.191059 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:23.191000    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 12:44:23.191271 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:23.191081    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 12:44:23.762995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:23.762392    4095 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 12:44:23.762995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:23.762462    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 12:44:24.177997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:24.176001    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 12:44:24.177997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:24.176066    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 12:44:24.177997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:24.176674    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 12:44:24.177997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:24.176709    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 12:44:24.769993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:24.759351    4095 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 12:44:24.769993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:24.759414    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 12:44:25.160997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:25.160280    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 12:44:25.160997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:25.160342    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 12:44:25.181096 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:25.181062    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 12:44:25.183297 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:25.183256    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 12:44:25.760005 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:25.759370    4095 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 12:44:25.760005 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:25.759433    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 12:44:26.163439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:26.163400    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 01 12:44:26.163706 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:26.163676    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 01 12:44:26.176100 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:26.176065    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 01 12:44:26.176337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:26.176311    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 01 12:44:26.759170 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:26.759125    4095 patch_prober.go:29] interesting pod/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 01 12:44:26.759692 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:26.759210    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 01 12:44:27.663484 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:27.663448    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:27.664362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:27.664337    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:27.664604 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:27.664585    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:44:27.698737 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:27.698700    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:27.699136 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:27.699094    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:27.700042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:27.700021    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:44:27.763310 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:27.763280    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:27.763787 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:27.763601    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:27.764542 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:27.764521    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:44:49.133059 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:44:49.133013    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:44:50.539136 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:50.539098    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:50.803056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:50.797384    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:44:50.839721 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:50.839675    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:50.840876 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:50.840842    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:44:50.841370 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:50.841345    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:50.845990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:50.841775    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:50.857016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:50.856075    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:50.889946 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:50.889901    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.119840 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.119072    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.165919 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.164423    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.224020 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.223951    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.264234 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.264134    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-sskbf" podUID="7ae7e690-ba05-45ab-9b7f-ea875d2e526a" containerName="cert-manager-webhook" probeResult="failure" output="Get \"http://10.134.0.23:6080/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.511054 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.510792    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.554128 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.554075    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.555181 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.555153    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:44:51.899997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.889841    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.899997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.890440    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.899997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.891125    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:44:51.899997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.891213    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.926573    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.927182    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.927735    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.927825    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.928540    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.929126    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.947132    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.947239    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.948467    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:51.959004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.948517    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.989002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.987552    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:51.993997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:51.990331    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:44:52.143001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.142783    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.147004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.144225    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:44:52.310052 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.307410    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.664086 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.633541    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.848997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.839348    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.896001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.876177    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Nov 01 12:44:52.935053 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.897264    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.938995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.916322    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.956996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.928100    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:52.978089 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.928191    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 12:44:53.000998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.932400    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:53.015134 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:52.932456    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:44:53.015478 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.015441    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 01 12:44:53.146243 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.146184    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:53.312328 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.312238    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 12:44:53.593164 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.593071    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:53.593435 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.593411    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Nov 01 12:44:53.641325 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.641276    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:53.711994 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:44:53.677344    4095 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.475s"
Nov 01 12:44:53.711994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.677434    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:44:53.711994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.677484    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:44:53.745639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.745597    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:53.872013 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:53.869758    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.153055 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.145067    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:54.153055 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.145133    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.185043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.175012    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 12:44:54.185043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.180925    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:44:54.185043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.183255    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 01 12:44:54.185043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.183548    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 01 12:44:54.185043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.183819    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 01 12:44:54.185043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.184094    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:44:54.185043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.184332    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.202652 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.202607    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 01 12:44:54.340537 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.340489    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.451061 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.441668    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:54.451061 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.441730    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.642683 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.642644    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:54.642996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.642947    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.646994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.643222    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.746147 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.746050    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:54.746728 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.746681    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:54.885604 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:54.885562    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.121992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.116843    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:44:55.121992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.116923    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.176104 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.176061    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.201130 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.201082    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.203159 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.201882    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.216605 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.203375    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.220994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.201943    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 01 12:44:55.220994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.201987    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.220994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.202046    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:44:55.220994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.202024    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 01 12:44:55.366469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.366427    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded"
Nov 01 12:44:55.420959 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.420856    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:44:55.525567 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.525536    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:44:55.577418 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.577383    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:44:55.626296 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.626267    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:44:55.647380 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.647255    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 12:44:55.732192 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.732111    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:44:55.748249 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.748224    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:44:55.788016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.786928    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:44:55.788016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.787005    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:44:55.788016 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.787041    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:44:55.793533 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.793188    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:44:55.793533 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.793461    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:44:55.794047 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:44:55.794026    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:45:17.043483 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.043448    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"]
Nov 01 12:45:17.048907 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.048881    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.130253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.130204    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97278f37-3f8b-4c52-9724-89148c474cb0-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.130253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.130260    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.130519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.130297    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.130519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.130389    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97278f37-3f8b-4c52-9724-89148c474cb0-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.130519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.130481    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.130655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.130631    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwfm\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-kube-api-access-4pwfm\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.130719 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.130697    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.130768 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.130724    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.130815 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.130766    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.159246 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.159159    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"]
Nov 01 12:45:17.233635 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.233602    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97278f37-3f8b-4c52-9724-89148c474cb0-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.233836 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.233644    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.235106 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.235079    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.235430 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.235349    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97278f37-3f8b-4c52-9724-89148c474cb0-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.235636 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.235619    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.235873 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.235858    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwfm\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-kube-api-access-4pwfm\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.236074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.236057    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.236421 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.236387    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.238554 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.238513    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.239740 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.239185    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97278f37-3f8b-4c52-9724-89148c474cb0-istio-podinfo\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.240225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.240204    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-data\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.240571 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.240554    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.240992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.237225    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97278f37-3f8b-4c52-9724-89148c474cb0-istiod-ca-cert\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.241114 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.240863    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-credential-socket\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.241201 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.240927    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-certs\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.248618 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.248592    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-envoy\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.295454 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.295414    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-istio-token\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.299439 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.299410    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwfm\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-kube-api-access-4pwfm\") pod \"e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") " pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.367805 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.367771    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:17.808278 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:45:17.808235    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97278f37_3f8b_4c52_9724_89148c474cb0.slice/crio-0bddb671e9f3727859c1c74c486745d9e0ccc967cfff2c4d0e085d614ab54809 WatchSource:0}: Error finding container 0bddb671e9f3727859c1c74c486745d9e0ccc967cfff2c4d0e085d614ab54809: Status 404 returned error can't find the container with id 0bddb671e9f3727859c1c74c486745d9e0ccc967cfff2c4d0e085d614ab54809
Nov 01 12:45:17.809298 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.809270    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"]
Nov 01 12:45:17.811015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.810964    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:45:17.811098 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.811047    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:45:17.811098 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:17.811076    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:45:18.636994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:18.636264    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" event={"ID":"97278f37-3f8b-4c52-9724-89148c474cb0","Type":"ContainerStarted","Data":"237fedc586856ebcf2d33a4fe9645415059233420ffe794f69b45b36563664c9"}
Nov 01 12:45:18.636994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:18.636340    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" event={"ID":"97278f37-3f8b-4c52-9724-89148c474cb0","Type":"ContainerStarted","Data":"0bddb671e9f3727859c1c74c486745d9e0ccc967cfff2c4d0e085d614ab54809"}
Nov 01 12:45:18.831261 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:18.823265    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" podStartSLOduration=2.823245069 podStartE2EDuration="2.823245069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:45:18.820404308 +0000 UTC m=+1378.738199978" watchObservedRunningTime="2026-11-01 12:45:18.823245069 +0000 UTC m=+1378.741040737"
Nov 01 12:45:19.146489 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:45:19.146452    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:45:19.368889 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:19.368859    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:19.370364 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:19.370342    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 12:45:19.370433 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:19.370400    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 12:45:20.368333 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:20.368296    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 12:45:20.368882 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:20.368349    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 12:45:21.368223 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:21.368191    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 12:45:21.368407 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:21.368240    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 12:45:22.368641 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:22.368601    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 12:45:22.369263 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:22.369235    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 12:45:23.368399 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:23.368370    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 12:45:23.368567 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:23.368424    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 12:45:24.368680 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:24.368643    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body=
Nov 01 12:45:24.369177 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:24.368701    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused"
Nov 01 12:45:25.372851 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:25.372820    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:25.373295 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:25.373281    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:25.373904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:25.373882    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:45:31.176244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.176127    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.177071 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.177031    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.177432 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.177397    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.231938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.231865    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.246100 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.246051    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.359993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.359085    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.537868 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.537770    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.656995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.653207    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 01 12:45:31.736350 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.736297    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:31.818058 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:31.806130    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:32.170003 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.166034    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:32.186002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.184608    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 01 12:45:32.186002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.185031    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:45:32.186558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.186293    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:32.186634 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.186601    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:45:32.257116 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.254382    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 12:45:32.274304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.274271    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:45:32.382497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.382457    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:32.396558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.396527    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:45:32.564056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.556218    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:32.587995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.585106    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:45:32.853056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.745048    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:45:32.853056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.745361    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:45:32.853056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.806628    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 01 12:45:32.853056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.832884    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:45:32.872996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.853772    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded"
Nov 01 12:45:32.873307 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:32.871389    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:33.209996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.193553    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 01 12:45:33.210700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.210657    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:45:33.225235 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.224145    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:45:33.225235 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.224890    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:33.234574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.234539    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:33.255431 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.255393    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:33.483938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.466872    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:33.602999 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.596069    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:33.602999 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.596182    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:33.832888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.832784    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:33.938652 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.839484    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded"
Nov 01 12:45:33.994998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.909224    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:34.013029 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.909427    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:45:34.040566 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.925182    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 01 12:45:34.073996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:33.925386    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": context deadline exceeded"
Nov 01 12:45:34.151204 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.151168    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:45:34.504073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.223079    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:45:34.504073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.503771    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:34.504073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.250110    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:34.504698 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.226246    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:45:34.504698 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.296637    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:34.504698 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.342107    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:34.521180 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.521126    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:34.804057 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.752948    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 01 12:45:34.895729 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.830857    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 01 12:45:34.949185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:34.855040    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:35.044213 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.044160    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:35.069470 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.068432    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded"
Nov 01 12:45:35.272007 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.033684    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:35.351875 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.351791    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:45:35.352201 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.352183    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:45:35.525656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.522111    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:35.547245 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.538884    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 01 12:45:35.547245 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.539184    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:45:35.547245 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.539433    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded"
Nov 01 12:45:35.547245 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.539602    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded"
Nov 01 12:45:35.807995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:35.806801    4095 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.015005 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.014455    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.298995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.172935    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.354498 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.174384    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 12:45:36.395992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.213415    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded"
Nov 01 12:45:36.398374 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.255501    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": context deadline exceeded"
Nov 01 12:45:36.446759 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.300671    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.549684 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.528161    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.658002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.536159    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-49sgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:45:36.658002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.549257    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.658002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.612281    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:36.671043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.549292    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded"
Nov 01 12:45:36.671043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.606373    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 12:45:36.671043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.660502    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:45:36.671043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.660568    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 12:45:36.671427 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.549315    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded"
Nov 01 12:45:36.995540 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:36.995494    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:45:37.045089 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:37.045056    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 12:45:37.293042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:37.292042    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:37.364996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:37.354422    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:37.423710 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:37.422715    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:45:37.445286 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:37.445255    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 12:45:37.490135 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:37.490101    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 12:45:37.596812 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:37.596723    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:45:37.614074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:37.613689    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:45:37.705105 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:45:37.705080    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:45:49.118074 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:45:49.118034    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:46:06.531153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.531118    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"]
Nov 01 12:46:06.535882 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.535861    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.608959 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.608915    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-metrics-certs\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.609186 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.609159    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.609323 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.609299    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6v98\" (UniqueName: \"kubernetes.io/projected/116ff48f-63c3-4a49-a8c4-7f7c6894a697-kube-api-access-h6v98\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.609411 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.609393    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116ff48f-63c3-4a49-a8c4-7f7c6894a697-openshift-service-ca\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.634218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.634185    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-83558e4e-serving-cert\""
Nov 01 12:46:06.640344 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.640316    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"]
Nov 01 12:46:06.710838 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.710807    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.711074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.710883    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6v98\" (UniqueName: \"kubernetes.io/projected/116ff48f-63c3-4a49-a8c4-7f7c6894a697-kube-api-access-h6v98\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.711074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.710945    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116ff48f-63c3-4a49-a8c4-7f7c6894a697-openshift-service-ca\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.711074 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:46:06.710991    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-serving-cert: secret "maas-api-e2e-ait-83558e4e-serving-cert" not found
Nov 01 12:46:06.711074 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:46:06.711064    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls podName:116ff48f-63c3-4a49-a8c4-7f7c6894a697 nodeName:}" failed. No retries permitted until 2026-11-01 12:46:07.211044347 +0000 UTC m=+1427.128840022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls") pod "maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk" (UID: "116ff48f-63c3-4a49-a8c4-7f7c6894a697") : secret "maas-api-e2e-ait-83558e4e-serving-cert" not found
Nov 01 12:46:06.711367 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.711220    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-metrics-certs\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.711673 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.711646    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116ff48f-63c3-4a49-a8c4-7f7c6894a697-openshift-service-ca\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.713656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.713635    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-metrics-certs\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:06.766216 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:06.766189    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6v98\" (UniqueName: \"kubernetes.io/projected/116ff48f-63c3-4a49-a8c4-7f7c6894a697-kube-api-access-h6v98\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:07.221308 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:07.221273    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:07.223820 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:07.223788    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls\") pod \"maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:07.451063 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:07.451032    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:07.825792 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:46:07.825757    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116ff48f_63c3_4a49_a8c4_7f7c6894a697.slice/crio-f5bd54e31a71b6f377130f91a1958b046440f4ef925e2326169439b7c51e7491 WatchSource:0}: Error finding container f5bd54e31a71b6f377130f91a1958b046440f4ef925e2326169439b7c51e7491: Status 404 returned error can't find the container with id f5bd54e31a71b6f377130f91a1958b046440f4ef925e2326169439b7c51e7491
Nov 01 12:46:07.826288 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:07.826257    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"]
Nov 01 12:46:08.099409 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:08.099317    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk" event={"ID":"116ff48f-63c3-4a49-a8c4-7f7c6894a697","Type":"ContainerStarted","Data":"f5bd54e31a71b6f377130f91a1958b046440f4ef925e2326169439b7c51e7491"}
Nov 01 12:46:09.111244 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:09.111202    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk" event={"ID":"116ff48f-63c3-4a49-a8c4-7f7c6894a697","Type":"ContainerStarted","Data":"44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643"}
Nov 01 12:46:09.111682 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:09.111275    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:09.267341 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:09.267290    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk" podStartSLOduration=2.331791597 podStartE2EDuration="3.267274498s" totalImagesPullingTime="935.482901ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:46:09.264330659 +0000 UTC m=+1429.182126325" watchObservedRunningTime="2026-11-01 12:46:09.267274498 +0000 UTC m=+1429.185070164"
Nov 01 12:46:15.124448 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:15.124419    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:46:18.124544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.124510    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"]
Nov 01 12:46:18.129269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.129245    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.179294 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.179270    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b2ccbb55-serving-cert\""
Nov 01 12:46:18.193142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.193114    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20611649-657d-4c97-b1f6-2f5d5e65a39f-openshift-service-ca\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.193319 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.193296    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-metrics-certs\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.193501 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.193482    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.193606 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.193589    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-278mk\" (UniqueName: \"kubernetes.io/projected/20611649-657d-4c97-b1f6-2f5d5e65a39f-kube-api-access-278mk\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.230408 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.230382    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"]
Nov 01 12:46:18.295318 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.295277    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20611649-657d-4c97-b1f6-2f5d5e65a39f-openshift-service-ca\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.295486 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.295428    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-metrics-certs\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.295610 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.295591    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.295706 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.295689    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-278mk\" (UniqueName: \"kubernetes.io/projected/20611649-657d-4c97-b1f6-2f5d5e65a39f-kube-api-access-278mk\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.295940 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:46:18.295919    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-serving-cert: secret "maas-api-e2e-derive-b2ccbb55-serving-cert" not found
Nov 01 12:46:18.296137 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:46:18.296112    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls podName:20611649-657d-4c97-b1f6-2f5d5e65a39f nodeName:}" failed. No retries permitted until 2026-11-01 12:46:18.796096546 +0000 UTC m=+1438.713892190 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls") pod "maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79" (UID: "20611649-657d-4c97-b1f6-2f5d5e65a39f") : secret "maas-api-e2e-derive-b2ccbb55-serving-cert" not found
Nov 01 12:46:18.296233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.296041    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20611649-657d-4c97-b1f6-2f5d5e65a39f-openshift-service-ca\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.298585 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.298559    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-metrics-certs\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.350839 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.350813    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-278mk\" (UniqueName: \"kubernetes.io/projected/20611649-657d-4c97-b1f6-2f5d5e65a39f-kube-api-access-278mk\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.804220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.804183    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:18.806699 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:18.806668    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls\") pod \"maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:19.043188 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:19.043154    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:19.124696 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:46:19.124652    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:46:19.415438 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:46:19.415406    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20611649_657d_4c97_b1f6_2f5d5e65a39f.slice/crio-66e52445e67a5811de71acf5a175edcc90918767510ab88f49d214947061e2f7 WatchSource:0}: Error finding container 66e52445e67a5811de71acf5a175edcc90918767510ab88f49d214947061e2f7: Status 404 returned error can't find the container with id 66e52445e67a5811de71acf5a175edcc90918767510ab88f49d214947061e2f7
Nov 01 12:46:19.415551 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:19.415465    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"]
Nov 01 12:46:20.191422 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:20.191390    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79" event={"ID":"20611649-657d-4c97-b1f6-2f5d5e65a39f","Type":"ContainerStarted","Data":"66e52445e67a5811de71acf5a175edcc90918767510ab88f49d214947061e2f7"}
Nov 01 12:46:21.197625 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:21.197585    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79" event={"ID":"20611649-657d-4c97-b1f6-2f5d5e65a39f","Type":"ContainerStarted","Data":"3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a"}
Nov 01 12:46:21.198021 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:21.197646    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:21.311470 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:21.311419    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79" podStartSLOduration=2.427651162 podStartE2EDuration="3.311404919s" totalImagesPullingTime="883.753757ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:46:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:46:21.307495276 +0000 UTC m=+1441.225290942" watchObservedRunningTime="2026-11-01 12:46:21.311404919 +0000 UTC m=+1441.229200582"
Nov 01 12:46:27.212102 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:46:27.212072    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:46:49.115071 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:46:49.115033    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:47:19.124340 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:47:19.124302    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:47:20.374618 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:47:20.374590    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:47:20.384705 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:47:20.384685    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:47:49.153643 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:47:49.153604    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:48:04.485014 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.484959    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"]
Nov 01 12:48:04.485536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.485190    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk" podUID="116ff48f-63c3-4a49-a8c4-7f7c6894a697" containerName="maas-api" containerID="cri-o://44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643" gracePeriod=30
Nov 01 12:48:04.824127 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.824101    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:48:04.856354 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.856323    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116ff48f-63c3-4a49-a8c4-7f7c6894a697-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116ff48f-63c3-4a49-a8c4-7f7c6894a697-openshift-service-ca\") pod \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") "
Nov 01 12:48:04.856608 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.856493    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-metrics-certs\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-metrics-certs\") pod \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") "
Nov 01 12:48:04.856726 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.856706    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116ff48f-63c3-4a49-a8c4-7f7c6894a697-openshift-service-ca" pod "116ff48f-63c3-4a49-a8c4-7f7c6894a697" (UID: "116ff48f-63c3-4a49-a8c4-7f7c6894a697"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:48:04.856785 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.856732    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116ff48f-63c3-4a49-a8c4-7f7c6894a697-kube-api-access-h6v98\" (UniqueName: \"kubernetes.io/projected/116ff48f-63c3-4a49-a8c4-7f7c6894a697-kube-api-access-h6v98\") pod \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") "
Nov 01 12:48:04.856847 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.856819    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls\") pod \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\" (UID: \"116ff48f-63c3-4a49-a8c4-7f7c6894a697\") "
Nov 01 12:48:04.857458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.857435    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/116ff48f-63c3-4a49-a8c4-7f7c6894a697-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:04.858707 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.858670    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-metrics-certs" pod "116ff48f-63c3-4a49-a8c4-7f7c6894a697" (UID: "116ff48f-63c3-4a49-a8c4-7f7c6894a697"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:48:04.858893 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.858868    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls" pod "116ff48f-63c3-4a49-a8c4-7f7c6894a697" (UID: "116ff48f-63c3-4a49-a8c4-7f7c6894a697"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:48:04.859190 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.859169    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116ff48f-63c3-4a49-a8c4-7f7c6894a697-kube-api-access-h6v98" pod "116ff48f-63c3-4a49-a8c4-7f7c6894a697" (UID: "116ff48f-63c3-4a49-a8c4-7f7c6894a697"). InnerVolumeSpecName "kube-api-access-h6v98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:48:04.958794 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.958760    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:04.958794 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.958795    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6v98\" (UniqueName: \"kubernetes.io/projected/116ff48f-63c3-4a49-a8c4-7f7c6894a697-kube-api-access-h6v98\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:04.959012 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:04.958812    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/116ff48f-63c3-4a49-a8c4-7f7c6894a697-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:05.005076 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:05.005046    4095 generic.go:356] "Generic (PLEG): container finished" podID="116ff48f-63c3-4a49-a8c4-7f7c6894a697" containerID="44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643" exitCode=0
Nov 01 12:48:05.005251 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:05.005102    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"
Nov 01 12:48:05.005251 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:05.005136    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk" event={"ID":"116ff48f-63c3-4a49-a8c4-7f7c6894a697","Type":"ContainerDied","Data":"44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643"}
Nov 01 12:48:05.005251 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:05.005190    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk" event={"ID":"116ff48f-63c3-4a49-a8c4-7f7c6894a697","Type":"ContainerDied","Data":"f5bd54e31a71b6f377130f91a1958b046440f4ef925e2326169439b7c51e7491"}
Nov 01 12:48:05.005251 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:05.005214    4095 scope.go:122] "RemoveContainer" containerID="44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643"
Nov 01 12:48:05.016046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:05.016026    4095 scope.go:122] "RemoveContainer" containerID="44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643"
Nov 01 12:48:05.016317 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:48:05.016299    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643\": container with ID starting with 44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643 not found: ID does not exist" containerID="44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643"
Nov 01 12:48:05.016387 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:05.016331    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643"} err="failed to get container status \"44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643\": rpc error: code = NotFound desc = could not find container \"44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643\": container with ID starting with 44ee8548466919bc8ce53d33af53b30bdf74a1ccca2d3d13283d00b1f3314643 not found: ID does not exist"
Nov 01 12:48:05.170574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:05.170541    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"]
Nov 01 12:48:05.174776 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:05.174730    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-83558e4e-7f544495c4-wsxrk"]
Nov 01 12:48:06.194468 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:06.194428    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116ff48f-63c3-4a49-a8c4-7f7c6894a697" path="/var/lib/kubelet/pods/116ff48f-63c3-4a49-a8c4-7f7c6894a697/volumes"
Nov 01 12:48:12.536505 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.536475    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"]
Nov 01 12:48:12.536899 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.536682    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79" podUID="20611649-657d-4c97-b1f6-2f5d5e65a39f" containerName="maas-api" containerID="cri-o://3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a" gracePeriod=30
Nov 01 12:48:12.872596 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.872567    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:48:12.949759 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.949725    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-metrics-certs\") pod \"20611649-657d-4c97-b1f6-2f5d5e65a39f\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") "
Nov 01 12:48:12.949963 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.949807    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20611649-657d-4c97-b1f6-2f5d5e65a39f-kube-api-access-278mk\" (UniqueName: \"kubernetes.io/projected/20611649-657d-4c97-b1f6-2f5d5e65a39f-kube-api-access-278mk\") pod \"20611649-657d-4c97-b1f6-2f5d5e65a39f\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") "
Nov 01 12:48:12.949963 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.949835    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls\") pod \"20611649-657d-4c97-b1f6-2f5d5e65a39f\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") "
Nov 01 12:48:12.949963 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.949865    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20611649-657d-4c97-b1f6-2f5d5e65a39f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20611649-657d-4c97-b1f6-2f5d5e65a39f-openshift-service-ca\") pod \"20611649-657d-4c97-b1f6-2f5d5e65a39f\" (UID: \"20611649-657d-4c97-b1f6-2f5d5e65a39f\") "
Nov 01 12:48:12.950352 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.950322    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20611649-657d-4c97-b1f6-2f5d5e65a39f-openshift-service-ca" pod "20611649-657d-4c97-b1f6-2f5d5e65a39f" (UID: "20611649-657d-4c97-b1f6-2f5d5e65a39f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:48:12.950964 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.950937    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20611649-657d-4c97-b1f6-2f5d5e65a39f-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:12.951932 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.951908    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-metrics-certs" pod "20611649-657d-4c97-b1f6-2f5d5e65a39f" (UID: "20611649-657d-4c97-b1f6-2f5d5e65a39f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:48:12.952140 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.952121    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20611649-657d-4c97-b1f6-2f5d5e65a39f-kube-api-access-278mk" pod "20611649-657d-4c97-b1f6-2f5d5e65a39f" (UID: "20611649-657d-4c97-b1f6-2f5d5e65a39f"). InnerVolumeSpecName "kube-api-access-278mk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:48:12.952140 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:12.952124    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls" pod "20611649-657d-4c97-b1f6-2f5d5e65a39f" (UID: "20611649-657d-4c97-b1f6-2f5d5e65a39f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:48:13.052909 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.052874    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:13.052909 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.052905    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-278mk\" (UniqueName: \"kubernetes.io/projected/20611649-657d-4c97-b1f6-2f5d5e65a39f-kube-api-access-278mk\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:13.052909 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.052916    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20611649-657d-4c97-b1f6-2f5d5e65a39f-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:48:13.143851 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.143814    4095 generic.go:356] "Generic (PLEG): container finished" podID="20611649-657d-4c97-b1f6-2f5d5e65a39f" containerID="3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a" exitCode=0
Nov 01 12:48:13.144080 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.143876    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"
Nov 01 12:48:13.144080 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.143905    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79" event={"ID":"20611649-657d-4c97-b1f6-2f5d5e65a39f","Type":"ContainerDied","Data":"3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a"}
Nov 01 12:48:13.144080 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.144020    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79" event={"ID":"20611649-657d-4c97-b1f6-2f5d5e65a39f","Type":"ContainerDied","Data":"66e52445e67a5811de71acf5a175edcc90918767510ab88f49d214947061e2f7"}
Nov 01 12:48:13.144080 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.144055    4095 scope.go:122] "RemoveContainer" containerID="3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a"
Nov 01 12:48:13.160234 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.160048    4095 scope.go:122] "RemoveContainer" containerID="3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a"
Nov 01 12:48:13.161249 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:48:13.161221    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a\": container with ID starting with 3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a not found: ID does not exist" containerID="3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a"
Nov 01 12:48:13.161382 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.161255    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a"} err="failed to get container status \"3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a\": rpc error: code = NotFound desc = could not find container \"3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a\": container with ID starting with 3fd80e2daa8cb27444122fbeddc3a934a4ec44969dd8951fcae4b233b64e630a not found: ID does not exist"
Nov 01 12:48:13.313463 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.309159    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"]
Nov 01 12:48:13.319314 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:13.319276    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b2ccbb55-6d667fb764-67d79"]
Nov 01 12:48:14.199438 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:14.199405    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20611649-657d-4c97-b1f6-2f5d5e65a39f" path="/var/lib/kubelet/pods/20611649-657d-4c97-b1f6-2f5d5e65a39f/volumes"
Nov 01 12:48:19.116094 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:48:19.116056    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:48:36.127049 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:36.123078    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-j62wc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:48:36.127049 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:36.123153    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.33:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:48:36.481234 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:36.481125    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-49sgk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:48:36.481482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:36.481449    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.34:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:48:49.117923 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:48:49.117884    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:48:58.878708 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:58.878672    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"]
Nov 01 12:48:58.879356 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:58.879341    4095 cpu_manager.go:414] "containerMap: removing container" podUID="116ff48f-63c3-4a49-a8c4-7f7c6894a697" containerName="maas-api"
Nov 01 12:48:58.879402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:58.879359    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116ff48f-63c3-4a49-a8c4-7f7c6894a697" containerName="maas-api"
Nov 01 12:48:58.879402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:58.879397    4095 cpu_manager.go:414] "containerMap: removing container" podUID="20611649-657d-4c97-b1f6-2f5d5e65a39f" containerName="maas-api"
Nov 01 12:48:58.879466 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:58.879402    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20611649-657d-4c97-b1f6-2f5d5e65a39f" containerName="maas-api"
Nov 01 12:48:58.879525 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:58.879515    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="20611649-657d-4c97-b1f6-2f5d5e65a39f" containerName="maas-api"
Nov 01 12:48:58.879572 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:58.879536    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="116ff48f-63c3-4a49-a8c4-7f7c6894a697" containerName="maas-api"
Nov 01 12:48:58.883596 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:58.883576    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:58.992988 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:58.978161    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-envoy\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.038057 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.001817    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-47feb945-openshift-default-dockercfg-5zcj7\""
Nov 01 12:48:59.038330 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.038304    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d01769-afe3-402d-999b-0ecc4a64814f-istiod-ca-cert\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.078305 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.078270    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-token\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.078883 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.078856    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v8c7\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-kube-api-access-4v8c7\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.079166 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.079142    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-data\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.079280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.079196    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-credential-socket\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.102992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.079328    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-certs\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.103335 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.103307    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-socket\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.103644 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.103620    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-podinfo\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.120367 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.120319    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"]
Nov 01 12:48:59.209042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.206255    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-envoy\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.209042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.206332    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d01769-afe3-402d-999b-0ecc4a64814f-istiod-ca-cert\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.209042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.206410    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-token\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.209042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.206435    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v8c7\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-kube-api-access-4v8c7\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.209042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.206540    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-data\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.209042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.206580    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-credential-socket\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.209042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.206691    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-certs\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.209042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.206762    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-socket\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.209042 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.206824    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-podinfo\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.210456 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.210426    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-envoy\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.210898 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.210875    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-podinfo\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.211103 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.211068    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d01769-afe3-402d-999b-0ecc4a64814f-istiod-ca-cert\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.211210 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.211169    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-data\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.211376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.211346    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-credential-socket\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.211591 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.211573    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-certs\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.211779 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.211760    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-socket\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.266737 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.264747    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v8c7\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-kube-api-access-4v8c7\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.280994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.276097    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-token\") pod \"e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") " pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.502035 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.501935    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:48:59.937242 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.937163    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"]
Nov 01 12:48:59.939032 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.938625    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:48:59.939032 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.938770    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:48:59.939032 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:48:59.938858    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:49:00.543772 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:00.543723    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" event={"ID":"a2d01769-afe3-402d-999b-0ecc4a64814f","Type":"ContainerStarted","Data":"6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea"}
Nov 01 12:49:00.543958 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:00.543788    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" event={"ID":"a2d01769-afe3-402d-999b-0ecc4a64814f","Type":"ContainerStarted","Data":"00a03004a704a9e764933bb651c87daa69ce816035c5168547fd303fffe1df53"}
Nov 01 12:49:00.657351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:00.657282    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" podStartSLOduration=2.657258097 podStartE2EDuration="2.657258097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:48:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:49:00.654869817 +0000 UTC m=+1600.572665483" watchObservedRunningTime="2026-11-01 12:49:00.657258097 +0000 UTC m=+1600.575053764"
Nov 01 12:49:01.502383 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:01.502353    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:49:01.508262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:01.508218    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:49:01.550707 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:01.550674    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:49:01.551693 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:01.551667    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:49:04.739347 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.739313    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"]
Nov 01 12:49:04.744130 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.744110    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.799532 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.799494    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"]
Nov 01 12:49:04.814015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.800571    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-metrics-certs\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.814015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.800650    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-openshift-service-ca\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.814015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.800788    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.814015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.801017    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8999l\" (UniqueName: \"kubernetes.io/projected/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-kube-api-access-8999l\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.840857 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.840829    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-61ffb38d-serving-cert\""
Nov 01 12:49:04.902515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.902482    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.902714 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:49:04.902640    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-serving-cert: secret "maas-api-e2e-migrate-61ffb38d-serving-cert" not found
Nov 01 12:49:04.902790 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:49:04.902722    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls podName:ce25f3d0-01e8-490f-9c95-0de72d8eb35f nodeName:}" failed. No retries permitted until 2026-11-01 12:49:05.402693017 +0000 UTC m=+1605.320488688 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls") pod "maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn" (UID: "ce25f3d0-01e8-490f-9c95-0de72d8eb35f") : secret "maas-api-e2e-migrate-61ffb38d-serving-cert" not found
Nov 01 12:49:04.902790 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.902744    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8999l\" (UniqueName: \"kubernetes.io/projected/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-kube-api-access-8999l\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.902906 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.902847    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-metrics-certs\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.902988 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.902917    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-openshift-service-ca\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.903627 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.903605    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-openshift-service-ca\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.905925 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.905895    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-metrics-certs\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:04.972041 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:04.971950    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8999l\" (UniqueName: \"kubernetes.io/projected/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-kube-api-access-8999l\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:05.411841 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:05.411793    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:05.414442 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:05.414424    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls\") pod \"maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:05.658642 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:05.658614    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:06.032312 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:06.032280    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"]
Nov 01 12:49:06.033196 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:49:06.033161    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce25f3d0_01e8_490f_9c95_0de72d8eb35f.slice/crio-55afa7982966641382078b4bfc6667abff42d8efb0d328a8b3f1d4008639bd0c WatchSource:0}: Error finding container 55afa7982966641382078b4bfc6667abff42d8efb0d328a8b3f1d4008639bd0c: Status 404 returned error can't find the container with id 55afa7982966641382078b4bfc6667abff42d8efb0d328a8b3f1d4008639bd0c
Nov 01 12:49:06.034520 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:06.034503    4095 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:49:06.586490 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:06.586448    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn" event={"ID":"ce25f3d0-01e8-490f-9c95-0de72d8eb35f","Type":"ContainerStarted","Data":"55afa7982966641382078b4bfc6667abff42d8efb0d328a8b3f1d4008639bd0c"}
Nov 01 12:49:07.595036 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:07.595000    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn" event={"ID":"ce25f3d0-01e8-490f-9c95-0de72d8eb35f","Type":"ContainerStarted","Data":"ab18962a7b491f93cf8fe2c07069ed87c4c02f3897bc1d4645dd6c8977b0c141"}
Nov 01 12:49:07.595502 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:07.595064    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:07.743400 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:07.743347    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn" podStartSLOduration=2.925486933 podStartE2EDuration="3.743332853s" totalImagesPullingTime="817.84592ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:49:07.741187916 +0000 UTC m=+1607.658983583" watchObservedRunningTime="2026-11-01 12:49:07.743332853 +0000 UTC m=+1607.661128544"
Nov 01 12:49:11.178195 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.178161    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"]
Nov 01 12:49:11.182542 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.182520    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.293997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.283631    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-f6b043-openshift-default-dockercfg-nmd46\""
Nov 01 12:49:11.309579 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.309468    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"]
Nov 01 12:49:11.319904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.319812    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-socket\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.319904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.319885    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c3640-96a4-4192-adf9-adc86d71d430-istiod-ca-cert\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.320160 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.319941    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-podinfo\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.320160 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.319995    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-data\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.320160 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.320087    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wms9h\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-kube-api-access-wms9h\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.320326 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.320278    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-token\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.320397 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.320377    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-certs\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.320577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.320554    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-credential-socket\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.320651 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.320634    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-envoy\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.422381 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.422344    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wms9h\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-kube-api-access-wms9h\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.422588 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.422451    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-token\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.422588 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.422530    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-certs\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.422722 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.422613    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-credential-socket\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.422722 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.422653    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-envoy\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.422828 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.422746    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-socket\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.422828 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.422792    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c3640-96a4-4192-adf9-adc86d71d430-istiod-ca-cert\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.422941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.422840    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-podinfo\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.422941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.422859    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-data\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.423140 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.423102    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-certs\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.423213 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.423135    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-credential-socket\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.423420 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.423401    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-socket\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.423512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.423490    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-data\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.423742 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.423723    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c3640-96a4-4192-adf9-adc86d71d430-istiod-ca-cert\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.425222 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.425198    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-envoy\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.425589 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.425573    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-podinfo\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.477380 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.477304    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wms9h\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-kube-api-access-wms9h\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.491019 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.480205    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-token\") pod \"e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") " pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.498690 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.497032    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:11.985277 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:49:11.985248    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b0c3640_96a4_4192_adf9_adc86d71d430.slice/crio-ef170580196abac26404e2874fdc4a171ab55b10a9b354b8f528918994b069e6 WatchSource:0}: Error finding container ef170580196abac26404e2874fdc4a171ab55b10a9b354b8f528918994b069e6: Status 404 returned error can't find the container with id ef170580196abac26404e2874fdc4a171ab55b10a9b354b8f528918994b069e6
Nov 01 12:49:11.985655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.985633    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"]
Nov 01 12:49:11.987805 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.987779    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:49:11.987862 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.987848    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:49:11.987904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:11.987892    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:49:12.652631 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:12.652592    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" event={"ID":"6b0c3640-96a4-4192-adf9-adc86d71d430","Type":"ContainerStarted","Data":"25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3"}
Nov 01 12:49:12.653074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:12.652652    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" event={"ID":"6b0c3640-96a4-4192-adf9-adc86d71d430","Type":"ContainerStarted","Data":"ef170580196abac26404e2874fdc4a171ab55b10a9b354b8f528918994b069e6"}
Nov 01 12:49:12.814210 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:12.814161    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" podStartSLOduration=1.814146357 podStartE2EDuration="1.814146357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:49:12.810665796 +0000 UTC m=+1612.728461462" watchObservedRunningTime="2026-11-01 12:49:12.814146357 +0000 UTC m=+1612.731942022"
Nov 01 12:49:13.499724 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:13.499685    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:13.509793 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:13.509763    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:13.611649 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:13.611621    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:49:13.660041 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:13.660013    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:13.661265 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:13.661237    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:49:19.118606 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:49:19.118567    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:49:49.116148 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:49:49.116105    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:49:58.387214 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.387175    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"]
Nov 01 12:49:58.393431 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.393401    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:49:58.443340 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.443296    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"]
Nov 01 12:49:58.470474 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.470433    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9496594-0329-4d04-afc7-7fa862ff4d3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4\" (UID: \"a9496594-0329-4d04-afc7-7fa862ff4d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:49:58.471030 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.470991    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlv59\" (UniqueName: \"kubernetes.io/projected/a9496594-0329-4d04-afc7-7fa862ff4d3b-kube-api-access-jlv59\") pod \"payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4\" (UID: \"a9496594-0329-4d04-afc7-7fa862ff4d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:49:58.485642 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.485607    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-0cd09f-dockercfg-9z8fk\""
Nov 01 12:49:58.485642 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.485632    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-0cd09f\""
Nov 01 12:49:58.572653 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.572610    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9496594-0329-4d04-afc7-7fa862ff4d3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4\" (UID: \"a9496594-0329-4d04-afc7-7fa862ff4d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:49:58.572846 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.572790    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlv59\" (UniqueName: \"kubernetes.io/projected/a9496594-0329-4d04-afc7-7fa862ff4d3b-kube-api-access-jlv59\") pod \"payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4\" (UID: \"a9496594-0329-4d04-afc7-7fa862ff4d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:49:58.573445 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.573425    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9496594-0329-4d04-afc7-7fa862ff4d3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4\" (UID: \"a9496594-0329-4d04-afc7-7fa862ff4d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:49:58.651838 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.651754    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlv59\" (UniqueName: \"kubernetes.io/projected/a9496594-0329-4d04-afc7-7fa862ff4d3b-kube-api-access-jlv59\") pod \"payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4\" (UID: \"a9496594-0329-4d04-afc7-7fa862ff4d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:49:58.716202 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:58.715814    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:49:59.128879 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:49:59.128852    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"]
Nov 01 12:49:59.129166 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:49:59.129141    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9496594_0329_4d04_afc7_7fa862ff4d3b.slice/crio-8707993e98b795e8229a89bb763a28abaf48a7c78cc30aa2a5e923a1b15cb9f2 WatchSource:0}: Error finding container 8707993e98b795e8229a89bb763a28abaf48a7c78cc30aa2a5e923a1b15cb9f2: Status 404 returned error can't find the container with id 8707993e98b795e8229a89bb763a28abaf48a7c78cc30aa2a5e923a1b15cb9f2
Nov 01 12:50:00.089844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:00.089803    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4" event={"ID":"a9496594-0329-4d04-afc7-7fa862ff4d3b","Type":"ContainerStarted","Data":"276373fa6b60cd1f49bb55e79bf38b63f119915050cbbc005416dc0c7943f321"}
Nov 01 12:50:00.090329 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:00.090053    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:50:00.090329 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:00.090096    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4" event={"ID":"a9496594-0329-4d04-afc7-7fa862ff4d3b","Type":"ContainerStarted","Data":"8707993e98b795e8229a89bb763a28abaf48a7c78cc30aa2a5e923a1b15cb9f2"}
Nov 01 12:50:00.197834 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:00.197786    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4" podStartSLOduration=2.197772417 podStartE2EDuration="2.197772417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:49:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:50:00.197657769 +0000 UTC m=+1660.115453434" watchObservedRunningTime="2026-11-01 12:50:00.197772417 +0000 UTC m=+1660.115568086"
Nov 01 12:50:11.099181 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:11.099147    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:50:19.186939 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:50:19.186890    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:50:19.427641 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.427603    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"]
Nov 01 12:50:19.441164 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.441086    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:19.496773 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.496730    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"]
Nov 01 12:50:19.565997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.547073    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-187d06ed-dockercfg-q66bg\""
Nov 01 12:50:19.566446 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.555005    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-187d06ed\""
Nov 01 12:50:19.589433 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.578556    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxrq8\" (UniqueName: \"kubernetes.io/projected/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-kube-api-access-bxrq8\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l\" (UID: \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:19.589433 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.578607    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l\" (UID: \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:19.624192 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.624152    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"]
Nov 01 12:50:19.632087 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.632030    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:19.660805 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.660775    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"]
Nov 01 12:50:19.680426 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.680394    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxrq8\" (UniqueName: \"kubernetes.io/projected/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-kube-api-access-bxrq8\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l\" (UID: \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:19.680426 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.680432    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l\" (UID: \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:19.681458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.681435    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l\" (UID: \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:19.734773 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.734699    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxrq8\" (UniqueName: \"kubernetes.io/projected/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-kube-api-access-bxrq8\") pod \"payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l\" (UID: \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:19.771050 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.771015    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:19.781843 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.781811    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkgm5\" (UniqueName: \"kubernetes.io/projected/427f5e28-9a88-40a4-bb71-4605262b5441-kube-api-access-dkgm5\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4\" (UID: \"427f5e28-9a88-40a4-bb71-4605262b5441\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:19.782239 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.782180    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427f5e28-9a88-40a4-bb71-4605262b5441-plugins-config-volume\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4\" (UID: \"427f5e28-9a88-40a4-bb71-4605262b5441\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:19.894990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.893633    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkgm5\" (UniqueName: \"kubernetes.io/projected/427f5e28-9a88-40a4-bb71-4605262b5441-kube-api-access-dkgm5\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4\" (UID: \"427f5e28-9a88-40a4-bb71-4605262b5441\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:19.894990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.894026    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427f5e28-9a88-40a4-bb71-4605262b5441-plugins-config-volume\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4\" (UID: \"427f5e28-9a88-40a4-bb71-4605262b5441\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:19.894990 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:19.894934    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427f5e28-9a88-40a4-bb71-4605262b5441-plugins-config-volume\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4\" (UID: \"427f5e28-9a88-40a4-bb71-4605262b5441\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:20.024033 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:20.023216    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkgm5\" (UniqueName: \"kubernetes.io/projected/427f5e28-9a88-40a4-bb71-4605262b5441-kube-api-access-dkgm5\") pod \"payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4\" (UID: \"427f5e28-9a88-40a4-bb71-4605262b5441\") " pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:20.265849 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:20.265814    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:20.390093 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:20.390059    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"]
Nov 01 12:50:20.391441 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:50:20.391413    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f6212b_9ad5_4da9_b4ce_1ae647bdbff4.slice/crio-5adfecf8a47b3761d1400bce440133fc060bc4e4adbe808d066209b632de2758 WatchSource:0}: Error finding container 5adfecf8a47b3761d1400bce440133fc060bc4e4adbe808d066209b632de2758: Status 404 returned error can't find the container with id 5adfecf8a47b3761d1400bce440133fc060bc4e4adbe808d066209b632de2758
Nov 01 12:50:20.639806 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:50:20.639777    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod427f5e28_9a88_40a4_bb71_4605262b5441.slice/crio-69d17730f410af369eada98cfe5e922dc6b4b8f32ccaede30ec0525e86586946 WatchSource:0}: Error finding container 69d17730f410af369eada98cfe5e922dc6b4b8f32ccaede30ec0525e86586946: Status 404 returned error can't find the container with id 69d17730f410af369eada98cfe5e922dc6b4b8f32ccaede30ec0525e86586946
Nov 01 12:50:20.639930 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:20.639910    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"]
Nov 01 12:50:21.265912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:21.265880    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4" event={"ID":"427f5e28-9a88-40a4-bb71-4605262b5441","Type":"ContainerStarted","Data":"10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9"}
Nov 01 12:50:21.266380 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:21.265928    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4" event={"ID":"427f5e28-9a88-40a4-bb71-4605262b5441","Type":"ContainerStarted","Data":"69d17730f410af369eada98cfe5e922dc6b4b8f32ccaede30ec0525e86586946"}
Nov 01 12:50:21.266380 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:21.265981    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:21.267596 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:21.267569    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l" event={"ID":"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4","Type":"ContainerStarted","Data":"c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776"}
Nov 01 12:50:21.267700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:21.267616    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l" event={"ID":"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4","Type":"ContainerStarted","Data":"5adfecf8a47b3761d1400bce440133fc060bc4e4adbe808d066209b632de2758"}
Nov 01 12:50:21.267700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:21.267652    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:21.376601 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:21.376547    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4" podStartSLOduration=2.376533532 podStartE2EDuration="2.376533532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:50:21.374254544 +0000 UTC m=+1681.292050210" watchObservedRunningTime="2026-11-01 12:50:21.376533532 +0000 UTC m=+1681.294329198"
Nov 01 12:50:21.510617 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:21.510540    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l" podStartSLOduration=2.5105199750000002 podStartE2EDuration="2.510519975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:50:21.507006026 +0000 UTC m=+1681.424801692" watchObservedRunningTime="2026-11-01 12:50:21.510519975 +0000 UTC m=+1681.428315645"
Nov 01 12:50:25.127503 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.127460    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"]
Nov 01 12:50:25.132305 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.132275    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.181982 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.181933    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-f6b043-serving-cert\""
Nov 01 12:50:25.235515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.235477    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"]
Nov 01 12:50:25.280913 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.280876    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.281144 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.281059    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b5bb58d-e902-484d-a456-17c5c88f464f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.281144 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.281134    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g45p6\" (UniqueName: \"kubernetes.io/projected/4b5bb58d-e902-484d-a456-17c5c88f464f-kube-api-access-g45p6\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.281278 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.281229    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.382994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.382894    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.383165 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.383080    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.383234 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.383180    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b5bb58d-e902-484d-a456-17c5c88f464f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.383290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.383243    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g45p6\" (UniqueName: \"kubernetes.io/projected/4b5bb58d-e902-484d-a456-17c5c88f464f-kube-api-access-g45p6\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.383290 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:50:25.383246    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-serving-cert: secret "maas-api-e2e-shared-a-w2-f6b043-serving-cert" not found
Nov 01 12:50:25.383404 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:50:25.383330    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls podName:4b5bb58d-e902-484d-a456-17c5c88f464f nodeName:}" failed. No retries permitted until 2026-11-01 12:50:25.883304963 +0000 UTC m=+1685.801100630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls") pod "maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw" (UID: "4b5bb58d-e902-484d-a456-17c5c88f464f") : secret "maas-api-e2e-shared-a-w2-f6b043-serving-cert" not found
Nov 01 12:50:25.383815 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.383798    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b5bb58d-e902-484d-a456-17c5c88f464f-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.385369 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.385349    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.444885 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.444855    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g45p6\" (UniqueName: \"kubernetes.io/projected/4b5bb58d-e902-484d-a456-17c5c88f464f-kube-api-access-g45p6\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.891494 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.891463    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:25.893866 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:25.893845    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:26.047681 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:26.047650    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:26.429831 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:26.429804    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"]
Nov 01 12:50:26.430346 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:50:26.430323    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b5bb58d_e902_484d_a456_17c5c88f464f.slice/crio-3ba7fe041e840de1f9ceab4609bb6bd4f31f76891d0cf01f51d13ccbb11f7c14 WatchSource:0}: Error finding container 3ba7fe041e840de1f9ceab4609bb6bd4f31f76891d0cf01f51d13ccbb11f7c14: Status 404 returned error can't find the container with id 3ba7fe041e840de1f9ceab4609bb6bd4f31f76891d0cf01f51d13ccbb11f7c14
Nov 01 12:50:27.313833 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:27.313799    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw" event={"ID":"4b5bb58d-e902-484d-a456-17c5c88f464f","Type":"ContainerStarted","Data":"3ba7fe041e840de1f9ceab4609bb6bd4f31f76891d0cf01f51d13ccbb11f7c14"}
Nov 01 12:50:28.322315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:28.322277    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw" event={"ID":"4b5bb58d-e902-484d-a456-17c5c88f464f","Type":"ContainerStarted","Data":"af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70"}
Nov 01 12:50:28.322717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:28.322361    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:28.435231 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:28.435101    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw" podStartSLOduration=2.504311847 podStartE2EDuration="3.435086679s" totalImagesPullingTime="930.774832ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:50:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:50:28.432442531 +0000 UTC m=+1688.350238198" watchObservedRunningTime="2026-11-01 12:50:28.435086679 +0000 UTC m=+1688.352882344"
Nov 01 12:50:32.277633 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:32.277529    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:50:32.277633 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:32.277593    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:50:34.336166 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:34.336138    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:50:47.592106 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:47.592071    4095 patch_prober.go:29] interesting pod/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:50:47.592631 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:47.592133    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:50:49.115456 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:50:49.115415    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:50:49.522245 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:49.522154    4095 generic.go:356] "Generic (PLEG): container finished" podID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerID="6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea" exitCode=0
Nov 01 12:50:49.522429 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:49.522231    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" event={"ID":"a2d01769-afe3-402d-999b-0ecc4a64814f","Type":"ContainerDied","Data":"6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea"}
Nov 01 12:50:49.522874 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:49.522847    4095 scope.go:122] "RemoveContainer" containerID="6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea"
Nov 01 12:50:49.524842 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:49.524808    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:50:49.525015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:49.524899    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:50:49.525015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:49.524947    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:50:50.538092 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:50.538055    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" event={"ID":"a2d01769-afe3-402d-999b-0ecc4a64814f","Type":"ContainerStarted","Data":"f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031"}
Nov 01 12:50:51.527079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:51.523769    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:50:51.527349 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:51.527259    4095 patch_prober.go:29] interesting pod/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 12:50:51.527349 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:51.527318    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 12:50:51.687960 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:51.687918    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"]
Nov 01 12:50:51.688465 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:51.688241    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy" containerID="cri-o://9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264" gracePeriod=30
Nov 01 12:50:52.523931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:52.523469    4095 patch_prober.go:29] interesting pod/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 12:50:52.523931 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:52.523538    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 12:50:53.520992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:53.515846    4095 patch_prober.go:29] interesting pod/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 12:50:53.520992 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:53.515916    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 12:50:54.516893 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:54.516856    4095 patch_prober.go:29] interesting pod/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 12:50:54.517266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:54.517228    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 12:50:55.503588 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:55.503546    4095 patch_prober.go:29] interesting pod/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Nov 01 12:50:55.504144 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:55.504081    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Nov 01 12:50:56.506855 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:56.506827    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:50:56.507377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:56.507356    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:50:56.508011 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:56.507990    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:50:56.526290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:56.526266    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:50:56.526395 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:56.526326    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:50:56.981717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:56.981692    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:50:57.073814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.073721    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-envoy\") pod \"3faf786a-e922-45d7-a7b1-922e2929650b\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") "
Nov 01 12:50:57.073814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.073795    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-certs\") pod \"3faf786a-e922-45d7-a7b1-922e2929650b\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") "
Nov 01 12:50:57.074116 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.073913    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-credential-socket\") pod \"3faf786a-e922-45d7-a7b1-922e2929650b\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") "
Nov 01 12:50:57.074116 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074096    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3faf786a-e922-45d7-a7b1-922e2929650b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faf786a-e922-45d7-a7b1-922e2929650b-istiod-ca-cert\") pod \"3faf786a-e922-45d7-a7b1-922e2929650b\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") "
Nov 01 12:50:57.074225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074108    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-certs" pod "3faf786a-e922-45d7-a7b1-922e2929650b" (UID: "3faf786a-e922-45d7-a7b1-922e2929650b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:57.074225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074152    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3faf786a-e922-45d7-a7b1-922e2929650b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faf786a-e922-45d7-a7b1-922e2929650b-istio-podinfo\") pod \"3faf786a-e922-45d7-a7b1-922e2929650b\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") "
Nov 01 12:50:57.074225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074215    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-credential-socket" pod "3faf786a-e922-45d7-a7b1-922e2929650b" (UID: "3faf786a-e922-45d7-a7b1-922e2929650b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:57.074378 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074281    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-data\") pod \"3faf786a-e922-45d7-a7b1-922e2929650b\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") "
Nov 01 12:50:57.074378 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074323    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-socket\") pod \"3faf786a-e922-45d7-a7b1-922e2929650b\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") "
Nov 01 12:50:57.074476 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074413    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-kube-api-access-w7qdn\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-kube-api-access-w7qdn\") pod \"3faf786a-e922-45d7-a7b1-922e2929650b\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") "
Nov 01 12:50:57.074530 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074480    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3faf786a-e922-45d7-a7b1-922e2929650b-istiod-ca-cert" pod "3faf786a-e922-45d7-a7b1-922e2929650b" (UID: "3faf786a-e922-45d7-a7b1-922e2929650b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:50:57.074530 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074480    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-istio-token\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-istio-token\") pod \"3faf786a-e922-45d7-a7b1-922e2929650b\" (UID: \"3faf786a-e922-45d7-a7b1-922e2929650b\") "
Nov 01 12:50:57.074651 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074585    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-socket" pod "3faf786a-e922-45d7-a7b1-922e2929650b" (UID: "3faf786a-e922-45d7-a7b1-922e2929650b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:57.074848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.074813    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-data" pod "3faf786a-e922-45d7-a7b1-922e2929650b" (UID: "3faf786a-e922-45d7-a7b1-922e2929650b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:57.075353 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.075334    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3faf786a-e922-45d7-a7b1-922e2929650b-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:57.075479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.075365    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:57.075479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.075376    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:57.075479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.075385    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:57.075479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.075394    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:57.076398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.076372    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3faf786a-e922-45d7-a7b1-922e2929650b-istio-podinfo" pod "3faf786a-e922-45d7-a7b1-922e2929650b" (UID: "3faf786a-e922-45d7-a7b1-922e2929650b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:50:57.076488 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.076434    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-envoy" pod "3faf786a-e922-45d7-a7b1-922e2929650b" (UID: "3faf786a-e922-45d7-a7b1-922e2929650b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:50:57.076983 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.076948    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-kube-api-access-w7qdn" pod "3faf786a-e922-45d7-a7b1-922e2929650b" (UID: "3faf786a-e922-45d7-a7b1-922e2929650b"). InnerVolumeSpecName "kube-api-access-w7qdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:57.077073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.077010    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-istio-token" pod "3faf786a-e922-45d7-a7b1-922e2929650b" (UID: "3faf786a-e922-45d7-a7b1-922e2929650b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:50:57.176938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.176897    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7qdn\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-kube-api-access-w7qdn\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:57.176938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.176930    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3faf786a-e922-45d7-a7b1-922e2929650b-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:57.176938 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.176944    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3faf786a-e922-45d7-a7b1-922e2929650b-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:57.177212 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.176957    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3faf786a-e922-45d7-a7b1-922e2929650b-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:50:57.595762 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.595730    4095 generic.go:356] "Generic (PLEG): container finished" podID="3faf786a-e922-45d7-a7b1-922e2929650b" containerID="9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264" exitCode=0
Nov 01 12:50:57.596275 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.595777    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"
Nov 01 12:50:57.596275 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.595809    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" event={"ID":"3faf786a-e922-45d7-a7b1-922e2929650b","Type":"ContainerDied","Data":"9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264"}
Nov 01 12:50:57.596275 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.595867    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn" event={"ID":"3faf786a-e922-45d7-a7b1-922e2929650b","Type":"ContainerDied","Data":"f8bfad03c985d03ba81783c98e3f4d0e27a9a6955d955c4823ba6a8e870d8442"}
Nov 01 12:50:57.596275 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.595897    4095 scope.go:122] "RemoveContainer" containerID="9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264"
Nov 01 12:50:57.607634 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.607609    4095 scope.go:122] "RemoveContainer" containerID="9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264"
Nov 01 12:50:57.607860 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:50:57.607838    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264\": container with ID starting with 9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264 not found: ID does not exist" containerID="9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264"
Nov 01 12:50:57.607910 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.607867    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264"} err="failed to get container status \"9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264\": rpc error: code = NotFound desc = could not find container \"9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264\": container with ID starting with 9f0db3e7fdc9bac41ae2d5303a65341a7cf295b2e0af6edea3140bec0264e264 not found: ID does not exist"
Nov 01 12:50:57.755337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.755308    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"]
Nov 01 12:50:57.802937 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:57.802905    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-95hqn"]
Nov 01 12:50:58.193817 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:50:58.193778    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" path="/var/lib/kubelet/pods/3faf786a-e922-45d7-a7b1-922e2929650b/volumes"
Nov 01 12:51:04.573139 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:04.573102    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"]
Nov 01 12:51:04.573646 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:04.573319    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy" containerID="cri-o://dc3b32a4b5a02fe62e6bc9ec74a31281826159d4f85e380f6f7261e7a05d2ad9" gracePeriod=30
Nov 01 12:51:06.359985 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:06.359939    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:51:06.360385 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:06.360011    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:51:09.703259 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:09.703217    4095 generic.go:356] "Generic (PLEG): container finished" podID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerID="dc3b32a4b5a02fe62e6bc9ec74a31281826159d4f85e380f6f7261e7a05d2ad9" exitCode=0
Nov 01 12:51:09.703634 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:09.703289    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" event={"ID":"030c9f49-c520-4f1b-b94e-fac2efd057b7","Type":"ContainerDied","Data":"dc3b32a4b5a02fe62e6bc9ec74a31281826159d4f85e380f6f7261e7a05d2ad9"}
Nov 01 12:51:09.863301 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:09.863269    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:51:10.008351 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.008313    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-token\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-token\") pod \"030c9f49-c520-4f1b-b94e-fac2efd057b7\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") "
Nov 01 12:51:10.008543 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.008367    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-envoy\") pod \"030c9f49-c520-4f1b-b94e-fac2efd057b7\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") "
Nov 01 12:51:10.008543 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.008419    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-kube-api-access-5q2fj\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-kube-api-access-5q2fj\") pod \"030c9f49-c520-4f1b-b94e-fac2efd057b7\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") "
Nov 01 12:51:10.008543 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.008486    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-data\") pod \"030c9f49-c520-4f1b-b94e-fac2efd057b7\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") "
Nov 01 12:51:10.008780 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.008591    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-podinfo\") pod \"030c9f49-c520-4f1b-b94e-fac2efd057b7\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") "
Nov 01 12:51:10.008780 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.008644    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-certs\") pod \"030c9f49-c520-4f1b-b94e-fac2efd057b7\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") "
Nov 01 12:51:10.008780 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.008763    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-credential-socket\") pod \"030c9f49-c520-4f1b-b94e-fac2efd057b7\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") "
Nov 01 12:51:10.008964 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.008922    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/030c9f49-c520-4f1b-b94e-fac2efd057b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/030c9f49-c520-4f1b-b94e-fac2efd057b7-istiod-ca-cert\") pod \"030c9f49-c520-4f1b-b94e-fac2efd057b7\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") "
Nov 01 12:51:10.009044 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.008987    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-socket\") pod \"030c9f49-c520-4f1b-b94e-fac2efd057b7\" (UID: \"030c9f49-c520-4f1b-b94e-fac2efd057b7\") "
Nov 01 12:51:10.009225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.009190    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-certs" pod "030c9f49-c520-4f1b-b94e-fac2efd057b7" (UID: "030c9f49-c520-4f1b-b94e-fac2efd057b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:10.009225 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.009208    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-data" pod "030c9f49-c520-4f1b-b94e-fac2efd057b7" (UID: "030c9f49-c520-4f1b-b94e-fac2efd057b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:10.009466 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.009431    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-credential-socket" pod "030c9f49-c520-4f1b-b94e-fac2efd057b7" (UID: "030c9f49-c520-4f1b-b94e-fac2efd057b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:10.009565 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.009497    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/030c9f49-c520-4f1b-b94e-fac2efd057b7-istiod-ca-cert" pod "030c9f49-c520-4f1b-b94e-fac2efd057b7" (UID: "030c9f49-c520-4f1b-b94e-fac2efd057b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:51:10.010253 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.010228    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.010368 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.010257    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.010368 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.010273    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.010368 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.010289    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/030c9f49-c520-4f1b-b94e-fac2efd057b7-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.010612 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.010460    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-socket" pod "030c9f49-c520-4f1b-b94e-fac2efd057b7" (UID: "030c9f49-c520-4f1b-b94e-fac2efd057b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:10.011270 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.011243    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-envoy" pod "030c9f49-c520-4f1b-b94e-fac2efd057b7" (UID: "030c9f49-c520-4f1b-b94e-fac2efd057b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:51:10.011810 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.011784    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-token" pod "030c9f49-c520-4f1b-b94e-fac2efd057b7" (UID: "030c9f49-c520-4f1b-b94e-fac2efd057b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:10.012356 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.012324    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-podinfo" pod "030c9f49-c520-4f1b-b94e-fac2efd057b7" (UID: "030c9f49-c520-4f1b-b94e-fac2efd057b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:51:10.012465 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.012444    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-kube-api-access-5q2fj" pod "030c9f49-c520-4f1b-b94e-fac2efd057b7" (UID: "030c9f49-c520-4f1b-b94e-fac2efd057b7"). InnerVolumeSpecName "kube-api-access-5q2fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:10.112089 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.112054    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.112089 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.112091    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.112290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.112106    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5q2fj\" (UniqueName: \"kubernetes.io/projected/030c9f49-c520-4f1b-b94e-fac2efd057b7-kube-api-access-5q2fj\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.112290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.112122    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/030c9f49-c520-4f1b-b94e-fac2efd057b7-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.112290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.112137    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/030c9f49-c520-4f1b-b94e-fac2efd057b7-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:10.712416 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.712386    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"
Nov 01 12:51:10.712416 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.712405    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj" event={"ID":"030c9f49-c520-4f1b-b94e-fac2efd057b7","Type":"ContainerDied","Data":"afcf8e372dcbe6df5fc57bc58eaeb59f460a0812b7cf339b6e3ec4a9992b96c6"}
Nov 01 12:51:10.712908 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.712494    4095 scope.go:122] "RemoveContainer" containerID="dc3b32a4b5a02fe62e6bc9ec74a31281826159d4f85e380f6f7261e7a05d2ad9"
Nov 01 12:51:10.920188 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.920160    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"]
Nov 01 12:51:10.967817 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:10.967727    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-wr2pj"]
Nov 01 12:51:12.192008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:12.191954    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" path="/var/lib/kubelet/pods/030c9f49-c520-4f1b-b94e-fac2efd057b7/volumes"
Nov 01 12:51:19.116839 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:51:19.116800    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:51:35.742353 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:35.742267    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"]
Nov 01 12:51:35.742811 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:35.742564    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn" podUID="ce25f3d0-01e8-490f-9c95-0de72d8eb35f" containerName="maas-api" containerID="cri-o://ab18962a7b491f93cf8fe2c07069ed87c4c02f3897bc1d4645dd6c8977b0c141" gracePeriod=30
Nov 01 12:51:35.897848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:35.897814    4095 generic.go:356] "Generic (PLEG): container finished" podID="ce25f3d0-01e8-490f-9c95-0de72d8eb35f" containerID="ab18962a7b491f93cf8fe2c07069ed87c4c02f3897bc1d4645dd6c8977b0c141" exitCode=0
Nov 01 12:51:35.898064 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:35.897887    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn" event={"ID":"ce25f3d0-01e8-490f-9c95-0de72d8eb35f","Type":"ContainerDied","Data":"ab18962a7b491f93cf8fe2c07069ed87c4c02f3897bc1d4645dd6c8977b0c141"}
Nov 01 12:51:36.086554 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.086520    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:51:36.191261 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.191222    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls\") pod \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") "
Nov 01 12:51:36.191444 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.191384    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-kube-api-access-8999l\" (UniqueName: \"kubernetes.io/projected/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-kube-api-access-8999l\") pod \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") "
Nov 01 12:51:36.191520 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.191471    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-openshift-service-ca\") pod \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") "
Nov 01 12:51:36.191692 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.191668    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-metrics-certs\") pod \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\" (UID: \"ce25f3d0-01e8-490f-9c95-0de72d8eb35f\") "
Nov 01 12:51:36.192966 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.192170    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-openshift-service-ca" pod "ce25f3d0-01e8-490f-9c95-0de72d8eb35f" (UID: "ce25f3d0-01e8-490f-9c95-0de72d8eb35f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:51:36.192966 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.192469    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:36.195046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.195015    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-metrics-certs" pod "ce25f3d0-01e8-490f-9c95-0de72d8eb35f" (UID: "ce25f3d0-01e8-490f-9c95-0de72d8eb35f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:51:36.195156 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.195103    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-kube-api-access-8999l" pod "ce25f3d0-01e8-490f-9c95-0de72d8eb35f" (UID: "ce25f3d0-01e8-490f-9c95-0de72d8eb35f"). InnerVolumeSpecName "kube-api-access-8999l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:51:36.197034 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.196951    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls" pod "ce25f3d0-01e8-490f-9c95-0de72d8eb35f" (UID: "ce25f3d0-01e8-490f-9c95-0de72d8eb35f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:51:36.295865 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.294044    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8999l\" (UniqueName: \"kubernetes.io/projected/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-kube-api-access-8999l\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:36.307337 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.307312    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:36.307608 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.307590    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ce25f3d0-01e8-490f-9c95-0de72d8eb35f-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:51:36.909455 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.909425    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"
Nov 01 12:51:36.909860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.909435    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn" event={"ID":"ce25f3d0-01e8-490f-9c95-0de72d8eb35f","Type":"ContainerDied","Data":"55afa7982966641382078b4bfc6667abff42d8efb0d328a8b3f1d4008639bd0c"}
Nov 01 12:51:36.909860 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:36.909573    4095 scope.go:122] "RemoveContainer" containerID="ab18962a7b491f93cf8fe2c07069ed87c4c02f3897bc1d4645dd6c8977b0c141"
Nov 01 12:51:37.068280 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:37.068247    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"]
Nov 01 12:51:37.118149 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:37.117806    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-61ffb38d-5f447d548-lhhmn"]
Nov 01 12:51:38.332543 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:38.332501    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce25f3d0-01e8-490f-9c95-0de72d8eb35f" path="/var/lib/kubelet/pods/ce25f3d0-01e8-490f-9c95-0de72d8eb35f/volumes"
Nov 01 12:51:49.140398 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:51:49.140355    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:51:50.367864 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.367804    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"]
Nov 01 12:51:50.369043 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.369020    4095 cpu_manager.go:414] "containerMap: removing container" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy"
Nov 01 12:51:50.369232 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.369214    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy"
Nov 01 12:51:50.369405 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.369393    4095 cpu_manager.go:414] "containerMap: removing container" podUID="ce25f3d0-01e8-490f-9c95-0de72d8eb35f" containerName="maas-api"
Nov 01 12:51:50.369488 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.369478    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce25f3d0-01e8-490f-9c95-0de72d8eb35f" containerName="maas-api"
Nov 01 12:51:50.369595 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.369586    4095 cpu_manager.go:414] "containerMap: removing container" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy"
Nov 01 12:51:50.369673 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.369662    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy"
Nov 01 12:51:50.369923 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.369909    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="3faf786a-e922-45d7-a7b1-922e2929650b" containerName="istio-proxy"
Nov 01 12:51:50.370068 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.370055    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce25f3d0-01e8-490f-9c95-0de72d8eb35f" containerName="maas-api"
Nov 01 12:51:50.370160 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.370150    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="030c9f49-c520-4f1b-b94e-fac2efd057b7" containerName="istio-proxy"
Nov 01 12:51:50.382022 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.377072    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.451325 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.451286    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgvrr\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-kube-api-access-lgvrr\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.453343 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.453314    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-certs\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.453564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.453544    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-token\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.455683 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.455661    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-envoy\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.455958 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.455937    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-podinfo\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.456304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.456284    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1064c601-a2a0-42af-9fcc-5886f08617e4-istiod-ca-cert\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.456481 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.456465    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-socket\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.456650 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.456631    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-data\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.456801 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.456786    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-credential-socket\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.465964 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.465927    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"]
Nov 01 12:51:50.496775 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.496533    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-c4bbec-openshift-default-dockercfg-kxwx4\""
Nov 01 12:51:50.572278 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.572084    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-podinfo\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.572278 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.572267    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1064c601-a2a0-42af-9fcc-5886f08617e4-istiod-ca-cert\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.572540 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.572331    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-socket\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.572540 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.572356    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-data\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.572540 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.572407    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-credential-socket\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.572540 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.572509    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgvrr\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-kube-api-access-lgvrr\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.572762 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.572548    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-certs\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.572762 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.572590    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-token\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.572762 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.572623    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-envoy\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.590082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.573600    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-data\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.590082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.576828    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1064c601-a2a0-42af-9fcc-5886f08617e4-istiod-ca-cert\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.590082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.581525    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-certs\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.590082 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.583618    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-podinfo\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.596573 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.596291    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-credential-socket\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.601907 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.601568    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-envoy\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.631041 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.628431    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-socket\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.661991 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.661367    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-token\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.664428 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.664399    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgvrr\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-kube-api-access-lgvrr\") pod \"e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") " pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:50.750809 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:50.750774    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:51.202571 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:51:51.201824    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1064c601_a2a0_42af_9fcc_5886f08617e4.slice/crio-34e8552d04f3c0d464f91655730c52f4170d15059238b77c63ff4832cd67700b WatchSource:0}: Error finding container 34e8552d04f3c0d464f91655730c52f4170d15059238b77c63ff4832cd67700b: Status 404 returned error can't find the container with id 34e8552d04f3c0d464f91655730c52f4170d15059238b77c63ff4832cd67700b
Nov 01 12:51:51.204434 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:51.203881    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"]
Nov 01 12:51:51.211476 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:51.211424    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:51:51.211640 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:51.211570    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:51:51.211797 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:51.211654    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:51:52.122572 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:52.121624    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" event={"ID":"1064c601-a2a0-42af-9fcc-5886f08617e4","Type":"ContainerStarted","Data":"e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620"}
Nov 01 12:51:52.122572 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:52.121699    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" event={"ID":"1064c601-a2a0-42af-9fcc-5886f08617e4","Type":"ContainerStarted","Data":"34e8552d04f3c0d464f91655730c52f4170d15059238b77c63ff4832cd67700b"}
Nov 01 12:51:52.282505 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:52.282456    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" podStartSLOduration=2.282441419 podStartE2EDuration="2.282441419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:51:52.233919011 +0000 UTC m=+1772.151714674" watchObservedRunningTime="2026-11-01 12:51:52.282441419 +0000 UTC m=+1772.200237140"
Nov 01 12:51:52.751138 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:52.751098    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:52.757846 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:52.757813    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:53.129579 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:53.129548    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:53.130684 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:53.130660    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:51:55.146622 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:55.146594    4095 generic.go:356] "Generic (PLEG): container finished" podID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerID="25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3" exitCode=0
Nov 01 12:51:55.147127 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:55.146669    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" event={"ID":"6b0c3640-96a4-4192-adf9-adc86d71d430","Type":"ContainerDied","Data":"25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3"}
Nov 01 12:51:55.147222 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:55.147205    4095 scope.go:122] "RemoveContainer" containerID="25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3"
Nov 01 12:51:55.148790 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:55.148758    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:51:55.148877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:55.148833    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:51:55.148877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:55.148865    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:51:56.156842 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:56.156802    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" event={"ID":"6b0c3640-96a4-4192-adf9-adc86d71d430","Type":"ContainerStarted","Data":"fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d"}
Nov 01 12:51:56.499524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:56.499426    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:51:56.500885 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:56.500862    4095 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:51:56.500998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:56.500909    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 12:51:57.571015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:57.566260    4095 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:51:57.571015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:57.566329    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 12:51:58.500375 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:58.500339    4095 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:51:58.500575 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:58.500406    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 12:51:58.660259 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:58.660227    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:51:59.500058 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:59.500024    4095 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:51:59.500239 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:51:59.500095    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 12:52:00.499785 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:52:00.499747    4095 patch_prober.go:29] interesting pod/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 01 12:52:00.500228 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:52:00.499804    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 01 12:52:01.504079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:52:01.504047    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:52:01.505328 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:52:01.505287    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:52:19.117076 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:52:19.117016    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:52:20.439076 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:52:20.439050    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:52:20.446918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:52:20.446896    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:52:49.115560 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:52:49.115518    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:53:19.141676 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:53:19.141636    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:53:35.418412 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.418379    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"]
Nov 01 12:53:35.418901 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.418555    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy" containerID="cri-o://8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c" gracePeriod=30
Nov 01 12:53:35.467617 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.467567    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"]
Nov 01 12:53:35.467831 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.467806    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy" containerID="cri-o://bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3" gracePeriod=30
Nov 01 12:53:35.471488 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.471462    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"]
Nov 01 12:53:35.471674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.471651    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy" containerID="cri-o://753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2" gracePeriod=30
Nov 01 12:53:35.678544 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.678451    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"]
Nov 01 12:53:35.679149 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.679111    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy" containerID="cri-o://ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe" gracePeriod=30
Nov 01 12:53:35.913940 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.913905    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"]
Nov 01 12:53:35.914191 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.914165    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy" containerID="cri-o://db36717e60686331532d72fbb54c14352d385452ca20218369a3b1cd10e17321" gracePeriod=30
Nov 01 12:53:35.918022 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.917994    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"]
Nov 01 12:53:35.918184 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:35.918166    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy" containerID="cri-o://237fedc586856ebcf2d33a4fe9645415059233420ffe794f69b45b36563664c9" gracePeriod=30
Nov 01 12:53:37.172735 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:37.172691    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:53:37.173185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:37.172745    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:53:38.371951 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:38.371919    4095 patch_prober.go:29] interesting pod/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:53:38.372407 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:38.371993    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:53:40.375522 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.375484    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:53:40.375926 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.375552    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:53:40.845667 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.845635    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:53:40.850222 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.850200    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:53:40.856547 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.856526    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:53:40.908468 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908386    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-token\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-token\") pod \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") "
Nov 01 12:53:40.908468 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908449    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-envoy\") pod \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") "
Nov 01 12:53:40.908661 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908490    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-data\") pod \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") "
Nov 01 12:53:40.908661 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908544    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-podinfo\") pod \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") "
Nov 01 12:53:40.908661 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908586    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-data\") pod \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") "
Nov 01 12:53:40.908661 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908614    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-envoy\") pod \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") "
Nov 01 12:53:40.908661 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908643    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-credential-socket\") pod \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") "
Nov 01 12:53:40.908826 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908675    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-socket\") pod \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") "
Nov 01 12:53:40.908826 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908737    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-certs\") pod \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") "
Nov 01 12:53:40.908826 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908772    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-envoy\") pod \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") "
Nov 01 12:53:40.908826 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908801    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-kube-api-access-whhh6\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-kube-api-access-whhh6\") pod \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") "
Nov 01 12:53:40.908944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908906    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istiod-ca-cert\") pod \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") "
Nov 01 12:53:40.909010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908940    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-data\") pod \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") "
Nov 01 12:53:40.909010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.908997    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-podinfo\") pod \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") "
Nov 01 12:53:40.909079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909052    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-socket\") pod \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") "
Nov 01 12:53:40.909116 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909104    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-certs\") pod \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") "
Nov 01 12:53:40.909150 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909138    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istiod-ca-cert\") pod \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") "
Nov 01 12:53:40.909247 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909232    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-podinfo\") pod \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") "
Nov 01 12:53:40.909334 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909317    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-certs\") pod \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") "
Nov 01 12:53:40.909399 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909384    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-kube-api-access-6mgdq\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-kube-api-access-6mgdq\") pod \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") "
Nov 01 12:53:40.909458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909424    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-token\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-token\") pod \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") "
Nov 01 12:53:40.909517 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909458    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-credential-socket\") pod \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") "
Nov 01 12:53:40.909517 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909489    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-credential-socket\") pod \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") "
Nov 01 12:53:40.909616 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909551    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-token\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-token\") pod \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") "
Nov 01 12:53:40.909616 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909608    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istiod-ca-cert\") pod \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\" (UID: \"0490afc7-ea75-4d2c-b87f-ad207c89dab2\") "
Nov 01 12:53:40.909711 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909655    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-socket\") pod \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\" (UID: \"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2\") "
Nov 01 12:53:40.909839 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.909822    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-kube-api-access-678ks\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-kube-api-access-678ks\") pod \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\" (UID: \"4d0c4249-1e7d-4550-b383-8b3e41ea491c\") "
Nov 01 12:53:40.910682 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.910654    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-data" pod "0490afc7-ea75-4d2c-b87f-ad207c89dab2" (UID: "0490afc7-ea75-4d2c-b87f-ad207c89dab2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.910816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.910801    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:40.911262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.911239    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-credential-socket" pod "0490afc7-ea75-4d2c-b87f-ad207c89dab2" (UID: "0490afc7-ea75-4d2c-b87f-ad207c89dab2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.911468 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.911440    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-data" pod "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" (UID: "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.911995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.911688    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-credential-socket" pod "4d0c4249-1e7d-4550-b383-8b3e41ea491c" (UID: "4d0c4249-1e7d-4550-b383-8b3e41ea491c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.912221 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.912188    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istiod-ca-cert" pod "0490afc7-ea75-4d2c-b87f-ad207c89dab2" (UID: "0490afc7-ea75-4d2c-b87f-ad207c89dab2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:53:40.912530 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.912506    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-certs" pod "0490afc7-ea75-4d2c-b87f-ad207c89dab2" (UID: "0490afc7-ea75-4d2c-b87f-ad207c89dab2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.913564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.913353    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-socket" pod "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" (UID: "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.913564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.913366    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-socket" pod "0490afc7-ea75-4d2c-b87f-ad207c89dab2" (UID: "0490afc7-ea75-4d2c-b87f-ad207c89dab2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.913814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.913788    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-credential-socket" pod "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" (UID: "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.914796 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.914535    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istiod-ca-cert" pod "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" (UID: "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:53:40.915758 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.915237    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-data" pod "4d0c4249-1e7d-4550-b383-8b3e41ea491c" (UID: "4d0c4249-1e7d-4550-b383-8b3e41ea491c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.915758 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.915472    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-certs" pod "4d0c4249-1e7d-4550-b383-8b3e41ea491c" (UID: "4d0c4249-1e7d-4550-b383-8b3e41ea491c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.916905 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.916468    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-socket" pod "4d0c4249-1e7d-4550-b383-8b3e41ea491c" (UID: "4d0c4249-1e7d-4550-b383-8b3e41ea491c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.916905 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.916707    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-certs" pod "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" (UID: "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.916905 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.916875    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istiod-ca-cert" pod "4d0c4249-1e7d-4550-b383-8b3e41ea491c" (UID: "4d0c4249-1e7d-4550-b383-8b3e41ea491c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:53:40.918422 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.918396    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-token" pod "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" (UID: "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:40.919668 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.919632    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-envoy" pod "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" (UID: "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.920308 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.920286    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-envoy" pod "0490afc7-ea75-4d2c-b87f-ad207c89dab2" (UID: "0490afc7-ea75-4d2c-b87f-ad207c89dab2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:40.920687 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.920661    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-podinfo" pod "4d0c4249-1e7d-4550-b383-8b3e41ea491c" (UID: "4d0c4249-1e7d-4550-b383-8b3e41ea491c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:53:40.920920 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.920887    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-kube-api-access-whhh6" pod "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" (UID: "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2"). InnerVolumeSpecName "kube-api-access-whhh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:40.922281 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.922249    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-kube-api-access-678ks" pod "4d0c4249-1e7d-4550-b383-8b3e41ea491c" (UID: "4d0c4249-1e7d-4550-b383-8b3e41ea491c"). InnerVolumeSpecName "kube-api-access-678ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:40.930492 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.930465    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-token" pod "4d0c4249-1e7d-4550-b383-8b3e41ea491c" (UID: "4d0c4249-1e7d-4550-b383-8b3e41ea491c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:40.930492 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.930417    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-podinfo" pod "0490afc7-ea75-4d2c-b87f-ad207c89dab2" (UID: "0490afc7-ea75-4d2c-b87f-ad207c89dab2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:53:40.930669 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.930645    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-podinfo" pod "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" (UID: "3bc7262d-f0eb-49c8-8a2f-f859df7de7f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:53:40.930788 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.930673    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-token" pod "0490afc7-ea75-4d2c-b87f-ad207c89dab2" (UID: "0490afc7-ea75-4d2c-b87f-ad207c89dab2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:40.930788 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.930693    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-kube-api-access-6mgdq" pod "0490afc7-ea75-4d2c-b87f-ad207c89dab2" (UID: "0490afc7-ea75-4d2c-b87f-ad207c89dab2"). InnerVolumeSpecName "kube-api-access-6mgdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:40.931827 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:40.931793    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-envoy" pod "4d0c4249-1e7d-4550-b383-8b3e41ea491c" (UID: "4d0c4249-1e7d-4550-b383-8b3e41ea491c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.012575 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012539    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-678ks\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-kube-api-access-678ks\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012575 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012569    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012575 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012580    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012589    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012598    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012606    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012615    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012624    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012632    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012642    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012650    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012658    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whhh6\" (UniqueName: \"kubernetes.io/projected/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-kube-api-access-whhh6\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012667    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012676    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012685    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012693    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0490afc7-ea75-4d2c-b87f-ad207c89dab2-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012700    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012708    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012716    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012724    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6mgdq\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-kube-api-access-6mgdq\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012733    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012744    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012756    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4d0c4249-1e7d-4550-b383-8b3e41ea491c-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012768    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4d0c4249-1e7d-4550-b383-8b3e41ea491c-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012780    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0490afc7-ea75-4d2c-b87f-ad207c89dab2-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.012891 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.012788    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.044200 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.044161    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:53:41.114457 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.113997    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09815128-1266-4fed-b6a1-d059cc352fc4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09815128-1266-4fed-b6a1-d059cc352fc4-istiod-ca-cert\") pod \"09815128-1266-4fed-b6a1-d059cc352fc4\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") "
Nov 01 12:53:41.114457 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.114069    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/09815128-1266-4fed-b6a1-d059cc352fc4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09815128-1266-4fed-b6a1-d059cc352fc4-istio-podinfo\") pod \"09815128-1266-4fed-b6a1-d059cc352fc4\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") "
Nov 01 12:53:41.114457 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.114326    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-socket\") pod \"09815128-1266-4fed-b6a1-d059cc352fc4\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") "
Nov 01 12:53:41.114457 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.114399    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-certs\") pod \"09815128-1266-4fed-b6a1-d059cc352fc4\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") "
Nov 01 12:53:41.114457 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.114453    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-envoy\") pod \"09815128-1266-4fed-b6a1-d059cc352fc4\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") "
Nov 01 12:53:41.114872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.114502    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-kube-api-access-dbhhl\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-kube-api-access-dbhhl\") pod \"09815128-1266-4fed-b6a1-d059cc352fc4\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") "
Nov 01 12:53:41.114872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.114559    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-credential-socket\") pod \"09815128-1266-4fed-b6a1-d059cc352fc4\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") "
Nov 01 12:53:41.114872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.114620    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-istio-token\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-istio-token\") pod \"09815128-1266-4fed-b6a1-d059cc352fc4\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") "
Nov 01 12:53:41.114872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.114683    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-data\") pod \"09815128-1266-4fed-b6a1-d059cc352fc4\" (UID: \"09815128-1266-4fed-b6a1-d059cc352fc4\") "
Nov 01 12:53:41.120239 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.115752    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-data" pod "09815128-1266-4fed-b6a1-d059cc352fc4" (UID: "09815128-1266-4fed-b6a1-d059cc352fc4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.120239 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.115998    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-socket" pod "09815128-1266-4fed-b6a1-d059cc352fc4" (UID: "09815128-1266-4fed-b6a1-d059cc352fc4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.120239 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.116203    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-certs" pod "09815128-1266-4fed-b6a1-d059cc352fc4" (UID: "09815128-1266-4fed-b6a1-d059cc352fc4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.120239 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.116235    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-credential-socket" pod "09815128-1266-4fed-b6a1-d059cc352fc4" (UID: "09815128-1266-4fed-b6a1-d059cc352fc4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.120239 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.116698    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09815128-1266-4fed-b6a1-d059cc352fc4-istiod-ca-cert" pod "09815128-1266-4fed-b6a1-d059cc352fc4" (UID: "09815128-1266-4fed-b6a1-d059cc352fc4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:53:41.129028 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.127082    4095 generic.go:356] "Generic (PLEG): container finished" podID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerID="bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3" exitCode=0
Nov 01 12:53:41.129028 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.127172    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" event={"ID":"4d0c4249-1e7d-4550-b383-8b3e41ea491c","Type":"ContainerDied","Data":"bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3"}
Nov 01 12:53:41.129028 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.127210    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7" event={"ID":"4d0c4249-1e7d-4550-b383-8b3e41ea491c","Type":"ContainerDied","Data":"05270db1c900bf18102d89ae947aff7345df92cf74f31e00b2fda09cb9b23661"}
Nov 01 12:53:41.129028 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.127231    4095 scope.go:122] "RemoveContainer" containerID="bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3"
Nov 01 12:53:41.129028 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.127347    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"
Nov 01 12:53:41.130865 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.130833    4095 generic.go:356] "Generic (PLEG): container finished" podID="97278f37-3f8b-4c52-9724-89148c474cb0" containerID="237fedc586856ebcf2d33a4fe9645415059233420ffe794f69b45b36563664c9" exitCode=0
Nov 01 12:53:41.131000 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.130941    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" event={"ID":"97278f37-3f8b-4c52-9724-89148c474cb0","Type":"ContainerDied","Data":"237fedc586856ebcf2d33a4fe9645415059233420ffe794f69b45b36563664c9"}
Nov 01 12:53:41.138687 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.138611    4095 generic.go:356] "Generic (PLEG): container finished" podID="09815128-1266-4fed-b6a1-d059cc352fc4" containerID="ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe" exitCode=0
Nov 01 12:53:41.138865 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.138847    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"
Nov 01 12:53:41.139316 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.139126    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" event={"ID":"09815128-1266-4fed-b6a1-d059cc352fc4","Type":"ContainerDied","Data":"ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe"}
Nov 01 12:53:41.139316 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.139182    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k" event={"ID":"09815128-1266-4fed-b6a1-d059cc352fc4","Type":"ContainerDied","Data":"a7f285f4e1872cfb660276dd5184b0cfe343b1d54227f007013b5d95dba2119a"}
Nov 01 12:53:41.139426 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.139337    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-istio-token" pod "09815128-1266-4fed-b6a1-d059cc352fc4" (UID: "09815128-1266-4fed-b6a1-d059cc352fc4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:41.139426 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.139401    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-envoy" pod "09815128-1266-4fed-b6a1-d059cc352fc4" (UID: "09815128-1266-4fed-b6a1-d059cc352fc4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.141766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.141687    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/09815128-1266-4fed-b6a1-d059cc352fc4-istio-podinfo" pod "09815128-1266-4fed-b6a1-d059cc352fc4" (UID: "09815128-1266-4fed-b6a1-d059cc352fc4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:53:41.144844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.142255    4095 generic.go:356] "Generic (PLEG): container finished" podID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerID="753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2" exitCode=0
Nov 01 12:53:41.144844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.142319    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"
Nov 01 12:53:41.144844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.142378    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" event={"ID":"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2","Type":"ContainerDied","Data":"753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2"}
Nov 01 12:53:41.144844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.142428    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv" event={"ID":"3bc7262d-f0eb-49c8-8a2f-f859df7de7f2","Type":"ContainerDied","Data":"80f3118854a29bc974fd38df0958f310d5e0f5d7a0ff73384fb04dd7cd928cb1"}
Nov 01 12:53:41.144844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.143019    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-kube-api-access-dbhhl" pod "09815128-1266-4fed-b6a1-d059cc352fc4" (UID: "09815128-1266-4fed-b6a1-d059cc352fc4"). InnerVolumeSpecName "kube-api-access-dbhhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:41.144844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.144078    4095 generic.go:356] "Generic (PLEG): container finished" podID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerID="8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c" exitCode=0
Nov 01 12:53:41.144844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.144188    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" event={"ID":"0490afc7-ea75-4d2c-b87f-ad207c89dab2","Type":"ContainerDied","Data":"8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c"}
Nov 01 12:53:41.144844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.144226    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m" event={"ID":"0490afc7-ea75-4d2c-b87f-ad207c89dab2","Type":"ContainerDied","Data":"fc4b4289478e79ca41a2db2ff8cae29311bd9c6cbb97387dd169b765e23b2ca6"}
Nov 01 12:53:41.144844 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.144292    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"
Nov 01 12:53:41.149286 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.149268    4095 scope.go:122] "RemoveContainer" containerID="bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3"
Nov 01 12:53:41.149736 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:53:41.149710    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3\": container with ID starting with bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3 not found: ID does not exist" containerID="bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3"
Nov 01 12:53:41.149858 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.149744    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3"} err="failed to get container status \"bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3\": rpc error: code = NotFound desc = could not find container \"bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3\": container with ID starting with bd28f64ec7a6b3c9d64bc3d038c1d89f723b3650d42affcda7e57c75d5ae1be3 not found: ID does not exist"
Nov 01 12:53:41.149858 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.149766    4095 scope.go:122] "RemoveContainer" containerID="ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe"
Nov 01 12:53:41.150578 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.150481    4095 generic.go:356] "Generic (PLEG): container finished" podID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerID="db36717e60686331532d72fbb54c14352d385452ca20218369a3b1cd10e17321" exitCode=0
Nov 01 12:53:41.150578 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.150512    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" event={"ID":"4958fe8c-32cc-41b5-a2e9-e81fbf866721","Type":"ContainerDied","Data":"db36717e60686331532d72fbb54c14352d385452ca20218369a3b1cd10e17321"}
Nov 01 12:53:41.162208 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.162188    4095 scope.go:122] "RemoveContainer" containerID="198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756"
Nov 01 12:53:41.173775 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.173752    4095 scope.go:122] "RemoveContainer" containerID="ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe"
Nov 01 12:53:41.174124 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:53:41.174097    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe\": container with ID starting with ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe not found: ID does not exist" containerID="ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe"
Nov 01 12:53:41.174209 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.174137    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe"} err="failed to get container status \"ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe\": rpc error: code = NotFound desc = could not find container \"ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe\": container with ID starting with ee884af2429773119a73c852ec078ac714b248a5b5940d9a393d8714d58441fe not found: ID does not exist"
Nov 01 12:53:41.174209 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.174158    4095 scope.go:122] "RemoveContainer" containerID="198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756"
Nov 01 12:53:41.174442 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:53:41.174418    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756\": container with ID starting with 198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756 not found: ID does not exist" containerID="198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756"
Nov 01 12:53:41.174492 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.174453    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756"} err="failed to get container status \"198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756\": rpc error: code = NotFound desc = could not find container \"198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756\": container with ID starting with 198df2dafc7ce8fa83d8f4f02852ec735d9bc70e6b1c53849251d5241e9ce756 not found: ID does not exist"
Nov 01 12:53:41.174492 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.174472    4095 scope.go:122] "RemoveContainer" containerID="753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2"
Nov 01 12:53:41.191181 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.191150    4095 scope.go:122] "RemoveContainer" containerID="e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e"
Nov 01 12:53:41.206510 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.206041    4095 scope.go:122] "RemoveContainer" containerID="753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2"
Nov 01 12:53:41.206510 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:53:41.206400    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2\": container with ID starting with 753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2 not found: ID does not exist" containerID="753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2"
Nov 01 12:53:41.206510 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.206441    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2"} err="failed to get container status \"753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2\": rpc error: code = NotFound desc = could not find container \"753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2\": container with ID starting with 753a280eed827cccda9e84f3255a02c6850196a0150c7c963119e13f9d766fc2 not found: ID does not exist"
Nov 01 12:53:41.206510 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.206464    4095 scope.go:122] "RemoveContainer" containerID="e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e"
Nov 01 12:53:41.206860 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:53:41.206814    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e\": container with ID starting with e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e not found: ID does not exist" containerID="e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e"
Nov 01 12:53:41.206933 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.206865    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e"} err="failed to get container status \"e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e\": rpc error: code = NotFound desc = could not find container \"e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e\": container with ID starting with e8c6f60c7f3f0acc0a841a688898a4b5fbe3136c12839866654a5e0d64a5c16e not found: ID does not exist"
Nov 01 12:53:41.206933 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.206887    4095 scope.go:122] "RemoveContainer" containerID="8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c"
Nov 01 12:53:41.222926 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.222881    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.223511 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.223480    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.223511 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.223508    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.223766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.223524    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbhhl\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-kube-api-access-dbhhl\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.223766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.223539    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.223766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.223552    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09815128-1266-4fed-b6a1-d059cc352fc4-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.223766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.223565    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/09815128-1266-4fed-b6a1-d059cc352fc4-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.223766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.223578    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/09815128-1266-4fed-b6a1-d059cc352fc4-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.223766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.223591    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/09815128-1266-4fed-b6a1-d059cc352fc4-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.245392 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.245364    4095 scope.go:122] "RemoveContainer" containerID="8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c"
Nov 01 12:53:41.248067 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:53:41.247988    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c\": container with ID starting with 8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c not found: ID does not exist" containerID="8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c"
Nov 01 12:53:41.248262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.248040    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c"} err="failed to get container status \"8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c\": rpc error: code = NotFound desc = could not find container \"8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c\": container with ID starting with 8832620a8e9af90ac68dfa06801936ae39d8e8d448e847c97f0fa264c184160c not found: ID does not exist"
Nov 01 12:53:41.266185 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.266158    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:53:41.292700 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.292670    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:53:41.325158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325124    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-data\") pod \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") "
Nov 01 12:53:41.325328 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325194    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-envoy\") pod \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") "
Nov 01 12:53:41.325328 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325248    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-certs\") pod \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") "
Nov 01 12:53:41.325328 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325282    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-kube-api-access-xm96m\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-kube-api-access-xm96m\") pod \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") "
Nov 01 12:53:41.325515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325330    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-credential-socket\") pod \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") "
Nov 01 12:53:41.325515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325389    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-token\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-token\") pod \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") "
Nov 01 12:53:41.325515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325437    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-socket\") pod \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") "
Nov 01 12:53:41.325515 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325442    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-data" pod "4958fe8c-32cc-41b5-a2e9-e81fbf866721" (UID: "4958fe8c-32cc-41b5-a2e9-e81fbf866721"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.325722 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325564    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-podinfo\") pod \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") "
Nov 01 12:53:41.325722 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325594    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istiod-ca-cert\") pod \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\" (UID: \"4958fe8c-32cc-41b5-a2e9-e81fbf866721\") "
Nov 01 12:53:41.325722 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325610    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-certs" pod "4958fe8c-32cc-41b5-a2e9-e81fbf866721" (UID: "4958fe8c-32cc-41b5-a2e9-e81fbf866721"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.325888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325734    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-credential-socket" pod "4958fe8c-32cc-41b5-a2e9-e81fbf866721" (UID: "4958fe8c-32cc-41b5-a2e9-e81fbf866721"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.325888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.325769    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-socket" pod "4958fe8c-32cc-41b5-a2e9-e81fbf866721" (UID: "4958fe8c-32cc-41b5-a2e9-e81fbf866721"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.326116 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.326087    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istiod-ca-cert" pod "4958fe8c-32cc-41b5-a2e9-e81fbf866721" (UID: "4958fe8c-32cc-41b5-a2e9-e81fbf866721"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:53:41.326300 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.326284    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.326386 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.326301    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.326386 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.326310    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.326386 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.326318    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.326386 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.326326    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.327792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.327767    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-kube-api-access-xm96m" pod "4958fe8c-32cc-41b5-a2e9-e81fbf866721" (UID: "4958fe8c-32cc-41b5-a2e9-e81fbf866721"). InnerVolumeSpecName "kube-api-access-xm96m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:41.327882 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.327872    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-envoy" pod "4958fe8c-32cc-41b5-a2e9-e81fbf866721" (UID: "4958fe8c-32cc-41b5-a2e9-e81fbf866721"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.328159 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.328132    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-podinfo" pod "4958fe8c-32cc-41b5-a2e9-e81fbf866721" (UID: "4958fe8c-32cc-41b5-a2e9-e81fbf866721"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:53:41.328159 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.328144    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-token" pod "4958fe8c-32cc-41b5-a2e9-e81fbf866721" (UID: "4958fe8c-32cc-41b5-a2e9-e81fbf866721"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:41.348549 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.348511    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"]
Nov 01 12:53:41.360752 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.360723    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-cq8lv"]
Nov 01 12:53:41.426893 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.426800    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-credential-socket\") pod \"97278f37-3f8b-4c52-9724-89148c474cb0\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") "
Nov 01 12:53:41.426893 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.426858    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-data\") pod \"97278f37-3f8b-4c52-9724-89148c474cb0\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") "
Nov 01 12:53:41.427469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427014    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-istio-token\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-istio-token\") pod \"97278f37-3f8b-4c52-9724-89148c474cb0\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") "
Nov 01 12:53:41.427469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427093    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-envoy\") pod \"97278f37-3f8b-4c52-9724-89148c474cb0\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") "
Nov 01 12:53:41.427469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427107    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-credential-socket" pod "97278f37-3f8b-4c52-9724-89148c474cb0" (UID: "97278f37-3f8b-4c52-9724-89148c474cb0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.427469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427139    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-socket\") pod \"97278f37-3f8b-4c52-9724-89148c474cb0\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") "
Nov 01 12:53:41.427469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427173    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/97278f37-3f8b-4c52-9724-89148c474cb0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97278f37-3f8b-4c52-9724-89148c474cb0-istio-podinfo\") pod \"97278f37-3f8b-4c52-9724-89148c474cb0\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") "
Nov 01 12:53:41.427469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427217    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97278f37-3f8b-4c52-9724-89148c474cb0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97278f37-3f8b-4c52-9724-89148c474cb0-istiod-ca-cert\") pod \"97278f37-3f8b-4c52-9724-89148c474cb0\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") "
Nov 01 12:53:41.427469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427172    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-data" pod "97278f37-3f8b-4c52-9724-89148c474cb0" (UID: "97278f37-3f8b-4c52-9724-89148c474cb0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.427469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427249    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-kube-api-access-4pwfm\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-kube-api-access-4pwfm\") pod \"97278f37-3f8b-4c52-9724-89148c474cb0\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") "
Nov 01 12:53:41.427469 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427468    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-certs\") pod \"97278f37-3f8b-4c52-9724-89148c474cb0\" (UID: \"97278f37-3f8b-4c52-9724-89148c474cb0\") "
Nov 01 12:53:41.427941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427638    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97278f37-3f8b-4c52-9724-89148c474cb0-istiod-ca-cert" pod "97278f37-3f8b-4c52-9724-89148c474cb0" (UID: "97278f37-3f8b-4c52-9724-89148c474cb0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:53:41.427941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427693    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-socket" pod "97278f37-3f8b-4c52-9724-89148c474cb0" (UID: "97278f37-3f8b-4c52-9724-89148c474cb0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.427941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.427820    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-certs" pod "97278f37-3f8b-4c52-9724-89148c474cb0" (UID: "97278f37-3f8b-4c52-9724-89148c474cb0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.428200 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.428178    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.428269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.428201    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.428269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.428211    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.428269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.428221    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/97278f37-3f8b-4c52-9724-89148c474cb0-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.428269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.428230    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xm96m\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-kube-api-access-xm96m\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.428269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.428240    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.428269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.428249    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.428269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.428257    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4958fe8c-32cc-41b5-a2e9-e81fbf866721-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.428269 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.428270    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.429315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.429292    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-istio-token" pod "97278f37-3f8b-4c52-9724-89148c474cb0" (UID: "97278f37-3f8b-4c52-9724-89148c474cb0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:41.429542 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.429520    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-kube-api-access-4pwfm" pod "97278f37-3f8b-4c52-9724-89148c474cb0" (UID: "97278f37-3f8b-4c52-9724-89148c474cb0"). InnerVolumeSpecName "kube-api-access-4pwfm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:53:41.429717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.429690    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-envoy" pod "97278f37-3f8b-4c52-9724-89148c474cb0" (UID: "97278f37-3f8b-4c52-9724-89148c474cb0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:53:41.429818 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.429693    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/97278f37-3f8b-4c52-9724-89148c474cb0-istio-podinfo" pod "97278f37-3f8b-4c52-9724-89148c474cb0" (UID: "97278f37-3f8b-4c52-9724-89148c474cb0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:53:41.457813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.457785    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"]
Nov 01 12:53:41.461757 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.461737    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-25j5m"]
Nov 01 12:53:41.529389 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.529360    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.529389 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.529385    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/97278f37-3f8b-4c52-9724-89148c474cb0-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.529389 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.529396    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/97278f37-3f8b-4c52-9724-89148c474cb0-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.529709 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.529406    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwfm\" (UniqueName: \"kubernetes.io/projected/97278f37-3f8b-4c52-9724-89148c474cb0-kube-api-access-4pwfm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:53:41.686403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.686267    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"]
Nov 01 12:53:41.706284 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.702284    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-2sgp7"]
Nov 01 12:53:41.797049 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.797003    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"]
Nov 01 12:53:41.800340 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:41.800319    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-k928k"]
Nov 01 12:53:42.161262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.161234    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"
Nov 01 12:53:42.161262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.161242    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv" event={"ID":"97278f37-3f8b-4c52-9724-89148c474cb0","Type":"ContainerDied","Data":"0bddb671e9f3727859c1c74c486745d9e0ccc967cfff2c4d0e085d614ab54809"}
Nov 01 12:53:42.161546 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.161337    4095 scope.go:122] "RemoveContainer" containerID="237fedc586856ebcf2d33a4fe9645415059233420ffe794f69b45b36563664c9"
Nov 01 12:53:42.167636 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.167606    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"
Nov 01 12:53:42.167636 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.167615    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg" event={"ID":"4958fe8c-32cc-41b5-a2e9-e81fbf866721","Type":"ContainerDied","Data":"5e706f83139d1d70984b004a99c8fbf574abd239c29d76b5ed25a98b89fcae15"}
Nov 01 12:53:42.173198 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.173179    4095 scope.go:122] "RemoveContainer" containerID="db36717e60686331532d72fbb54c14352d385452ca20218369a3b1cd10e17321"
Nov 01 12:53:42.191834 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.191805    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" path="/var/lib/kubelet/pods/0490afc7-ea75-4d2c-b87f-ad207c89dab2/volumes"
Nov 01 12:53:42.192356 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.192341    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" path="/var/lib/kubelet/pods/09815128-1266-4fed-b6a1-d059cc352fc4/volumes"
Nov 01 12:53:42.192786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.192774    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" path="/var/lib/kubelet/pods/3bc7262d-f0eb-49c8-8a2f-f859df7de7f2/volumes"
Nov 01 12:53:42.193493 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.193477    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" path="/var/lib/kubelet/pods/4d0c4249-1e7d-4550-b383-8b3e41ea491c/volumes"
Nov 01 12:53:42.323019 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.322985    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"]
Nov 01 12:53:42.329925 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.329898    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-77srg"]
Nov 01 12:53:42.434897 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.434812    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"]
Nov 01 12:53:42.440467 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:42.440432    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-kqbhv"]
Nov 01 12:53:44.191219 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:44.191190    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" path="/var/lib/kubelet/pods/4958fe8c-32cc-41b5-a2e9-e81fbf866721/volumes"
Nov 01 12:53:44.191715 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:53:44.191701    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" path="/var/lib/kubelet/pods/97278f37-3f8b-4c52-9724-89148c474cb0/volumes"
Nov 01 12:53:49.115146 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:53:49.115106    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:54:19.137522 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:54:19.137487    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:54:20.200459 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:20.200424    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"]
Nov 01 12:54:20.200997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:20.200621    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy" containerID="cri-o://f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031" gracePeriod=30
Nov 01 12:54:24.131108 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:24.131068    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:54:24.131573 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:24.131128    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:54:25.414768 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.414737    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:54:25.495961 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.495874    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-socket\") pod \"a2d01769-afe3-402d-999b-0ecc4a64814f\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") "
Nov 01 12:54:25.495961 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.495934    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2d01769-afe3-402d-999b-0ecc4a64814f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d01769-afe3-402d-999b-0ecc4a64814f-istiod-ca-cert\") pod \"a2d01769-afe3-402d-999b-0ecc4a64814f\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") "
Nov 01 12:54:25.496266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496131    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-data\") pod \"a2d01769-afe3-402d-999b-0ecc4a64814f\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") "
Nov 01 12:54:25.496266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496183    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-kube-api-access-4v8c7\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-kube-api-access-4v8c7\") pod \"a2d01769-afe3-402d-999b-0ecc4a64814f\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") "
Nov 01 12:54:25.496266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496211    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-socket" pod "a2d01769-afe3-402d-999b-0ecc4a64814f" (UID: "a2d01769-afe3-402d-999b-0ecc4a64814f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:25.496266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496219    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-token\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-token\") pod \"a2d01769-afe3-402d-999b-0ecc4a64814f\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") "
Nov 01 12:54:25.496499 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496340    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-podinfo\") pod \"a2d01769-afe3-402d-999b-0ecc4a64814f\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") "
Nov 01 12:54:25.496499 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496360    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2d01769-afe3-402d-999b-0ecc4a64814f-istiod-ca-cert" pod "a2d01769-afe3-402d-999b-0ecc4a64814f" (UID: "a2d01769-afe3-402d-999b-0ecc4a64814f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:54:25.496499 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496387    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-data" pod "a2d01769-afe3-402d-999b-0ecc4a64814f" (UID: "a2d01769-afe3-402d-999b-0ecc4a64814f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:25.496499 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496457    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-credential-socket\") pod \"a2d01769-afe3-402d-999b-0ecc4a64814f\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") "
Nov 01 12:54:25.496716 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496504    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-certs\") pod \"a2d01769-afe3-402d-999b-0ecc4a64814f\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") "
Nov 01 12:54:25.496716 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496531    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-envoy\") pod \"a2d01769-afe3-402d-999b-0ecc4a64814f\" (UID: \"a2d01769-afe3-402d-999b-0ecc4a64814f\") "
Nov 01 12:54:25.496824 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496739    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-credential-socket" pod "a2d01769-afe3-402d-999b-0ecc4a64814f" (UID: "a2d01769-afe3-402d-999b-0ecc4a64814f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:25.496944 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.496897    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-certs" pod "a2d01769-afe3-402d-999b-0ecc4a64814f" (UID: "a2d01769-afe3-402d-999b-0ecc4a64814f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:25.497228 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.497208    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:25.497314 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.497229    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:25.497314 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.497239    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:25.497314 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.497248    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a2d01769-afe3-402d-999b-0ecc4a64814f-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:25.497314 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.497256    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:25.499022 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.498996    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-envoy" pod "a2d01769-afe3-402d-999b-0ecc4a64814f" (UID: "a2d01769-afe3-402d-999b-0ecc4a64814f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:25.499022 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.499007    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-kube-api-access-4v8c7" pod "a2d01769-afe3-402d-999b-0ecc4a64814f" (UID: "a2d01769-afe3-402d-999b-0ecc4a64814f"). InnerVolumeSpecName "kube-api-access-4v8c7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:25.499164 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.499045    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-token" pod "a2d01769-afe3-402d-999b-0ecc4a64814f" (UID: "a2d01769-afe3-402d-999b-0ecc4a64814f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:25.499164 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.499102    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-podinfo" pod "a2d01769-afe3-402d-999b-0ecc4a64814f" (UID: "a2d01769-afe3-402d-999b-0ecc4a64814f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:54:25.510873 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.510849    4095 generic.go:356] "Generic (PLEG): container finished" podID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerID="f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031" exitCode=0
Nov 01 12:54:25.511080 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.511061    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" event={"ID":"a2d01769-afe3-402d-999b-0ecc4a64814f","Type":"ContainerDied","Data":"f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031"}
Nov 01 12:54:25.511161 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.511147    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv" event={"ID":"a2d01769-afe3-402d-999b-0ecc4a64814f","Type":"ContainerDied","Data":"00a03004a704a9e764933bb651c87daa69ce816035c5168547fd303fffe1df53"}
Nov 01 12:54:25.511216 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.511182    4095 scope.go:122] "RemoveContainer" containerID="f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031"
Nov 01 12:54:25.511344 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.511329    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"
Nov 01 12:54:25.537055 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.537019    4095 scope.go:122] "RemoveContainer" containerID="6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea"
Nov 01 12:54:25.546536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.546519    4095 scope.go:122] "RemoveContainer" containerID="f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031"
Nov 01 12:54:25.546824 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:54:25.546804    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031\": container with ID starting with f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031 not found: ID does not exist" containerID="f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031"
Nov 01 12:54:25.546916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.546831    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031"} err="failed to get container status \"f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031\": rpc error: code = NotFound desc = could not find container \"f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031\": container with ID starting with f787c4bb632e5bd8c66251fcc5e4e6f7f3f552b5cb54f0975a67587492308031 not found: ID does not exist"
Nov 01 12:54:25.546916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.546850    4095 scope.go:122] "RemoveContainer" containerID="6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea"
Nov 01 12:54:25.547116 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:54:25.547096    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea\": container with ID starting with 6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea not found: ID does not exist" containerID="6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea"
Nov 01 12:54:25.547177 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.547125    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea"} err="failed to get container status \"6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea\": rpc error: code = NotFound desc = could not find container \"6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea\": container with ID starting with 6d6554647aaec2da8123e287ac025b0c0bce5ad5cc4f9759f9c0c5fca493aeea not found: ID does not exist"
Nov 01 12:54:25.598567 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.598527    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:25.598567 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.598568    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4v8c7\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-kube-api-access-4v8c7\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:25.598829 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.598585    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:25.598829 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.598599    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a2d01769-afe3-402d-999b-0ecc4a64814f-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:25.682900 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.682861    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"]
Nov 01 12:54:25.686014 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:25.685993    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-47feb945-openshift-default-66c5b87697-kt4jv"]
Nov 01 12:54:26.193399 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:26.193363    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" path="/var/lib/kubelet/pods/a2d01769-afe3-402d-999b-0ecc4a64814f/volumes"
Nov 01 12:54:26.521165 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:26.521131    4095 generic.go:356] "Generic (PLEG): container finished" podID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerID="e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620" exitCode=0
Nov 01 12:54:26.521615 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:26.521186    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" event={"ID":"1064c601-a2a0-42af-9fcc-5886f08617e4","Type":"ContainerDied","Data":"e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620"}
Nov 01 12:54:26.521692 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:26.521653    4095 scope.go:122] "RemoveContainer" containerID="e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620"
Nov 01 12:54:26.523312 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:26.523283    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:54:26.523401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:26.523352    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:54:26.523401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:26.523384    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:54:27.543475 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:27.543438    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" event={"ID":"1064c601-a2a0-42af-9fcc-5886f08617e4","Type":"ContainerStarted","Data":"15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c"}
Nov 01 12:54:27.751176 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:27.751139    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:54:27.751355 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:27.751333    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 12:54:27.751402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:27.751383    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 12:54:28.751632 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:28.751595    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 12:54:28.752108 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:28.751649    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 12:54:29.751358 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:29.751321    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 12:54:29.751526 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:29.751373    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 12:54:30.751879 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:30.751835    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 12:54:30.752430 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:30.751902    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 12:54:31.781886 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:31.781857    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body=
Nov 01 12:54:31.782525 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:31.782063    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused"
Nov 01 12:54:32.756799 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:32.756767    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:54:32.757217 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:32.757194    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:54:32.757837 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:32.757817    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:54:38.275564 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:38.275505    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"]
Nov 01 12:54:38.276107 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:38.275715    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy" containerID="cri-o://48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77" gracePeriod=30
Nov 01 12:54:39.876834 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.876805    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"]
Nov 01 12:54:39.877512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877494    4095 cpu_manager.go:414] "containerMap: removing container" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy"
Nov 01 12:54:39.877557 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877514    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy"
Nov 01 12:54:39.877557 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877527    4095 cpu_manager.go:414] "containerMap: removing container" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy"
Nov 01 12:54:39.877557 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877533    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy"
Nov 01 12:54:39.877557 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877540    4095 cpu_manager.go:414] "containerMap: removing container" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy"
Nov 01 12:54:39.877557 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877545    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy"
Nov 01 12:54:39.877557 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877554    4095 cpu_manager.go:414] "containerMap: removing container" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy"
Nov 01 12:54:39.877557 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877559    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877574    4095 cpu_manager.go:414] "containerMap: removing container" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877579    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877588    4095 cpu_manager.go:414] "containerMap: removing container" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877593    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877602    4095 cpu_manager.go:414] "containerMap: removing container" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877606    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877614    4095 cpu_manager.go:414] "containerMap: removing container" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877618    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877645    4095 cpu_manager.go:414] "containerMap: removing container" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877650    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877739    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="4958fe8c-32cc-41b5-a2e9-e81fbf866721" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877750    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877755    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877763    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy"
Nov 01 12:54:39.877766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877768    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="97278f37-3f8b-4c52-9724-89148c474cb0" containerName="istio-proxy"
Nov 01 12:54:39.878237 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877776    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bc7262d-f0eb-49c8-8a2f-f859df7de7f2" containerName="istio-proxy"
Nov 01 12:54:39.878237 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877783    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d0c4249-1e7d-4550-b383-8b3e41ea491c" containerName="istio-proxy"
Nov 01 12:54:39.878237 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877789    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="0490afc7-ea75-4d2c-b87f-ad207c89dab2" containerName="istio-proxy"
Nov 01 12:54:39.878237 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.877795    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="09815128-1266-4fed-b6a1-d059cc352fc4" containerName="istio-proxy"
Nov 01 12:54:39.881833 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.881817    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:39.937993 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:39.937316    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-78c69d-openshift-default-dockercfg-pp9lt\""
Nov 01 12:54:40.021001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.015053    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45zgl\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-kube-api-access-45zgl\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.021001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.015107    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-certs\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.021001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.015200    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-envoy\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.021001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.015229    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istiod-ca-cert\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.021001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.015286    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-data\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.021001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.015348    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-podinfo\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.021001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.015375    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-token\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.021001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.015462    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-credential-socket\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.021001 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.015540    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-socket\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.077415 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.077379    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"]
Nov 01 12:54:40.118789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.118741    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45zgl\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-kube-api-access-45zgl\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.119032 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.118792    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-certs\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.119032 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.118902    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-envoy\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.119032 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.118930    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istiod-ca-cert\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.119207 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.119063    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-data\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.119207 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.119137    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-podinfo\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.119207 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.119167    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-token\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.119369 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.119266    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-credential-socket\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.119369 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.119353    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-socket\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.119999 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.119959    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-socket\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.120247 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.120228    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-data\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.122941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.120624    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istiod-ca-cert\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.122941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.121164    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-certs\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.122941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.122132    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-credential-socket\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.131315 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.131218    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-podinfo\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.131479 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.131454    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-envoy\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.175180 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.175141    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45zgl\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-kube-api-access-45zgl\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.175338 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.175204    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-token\") pod \"e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") " pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.208180 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.208144    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:40.639786 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:54:40.639757    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dc78e78_3863_44a7_b0b1_5f96566e8ec8.slice/crio-7806faa5cb63f2e2982dc80e862137dc5b925df3ab995f0f4dc7115492a4c4a5 WatchSource:0}: Error finding container 7806faa5cb63f2e2982dc80e862137dc5b925df3ab995f0f4dc7115492a4c4a5: Status 404 returned error can't find the container with id 7806faa5cb63f2e2982dc80e862137dc5b925df3ab995f0f4dc7115492a4c4a5
Nov 01 12:54:40.640076 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.640052    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"]
Nov 01 12:54:40.642789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.642762    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:54:40.642852 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.642821    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:54:40.642885 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.642850    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:54:40.763204 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.763163    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" event={"ID":"4dc78e78-3863-44a7-b0b1-5f96566e8ec8","Type":"ContainerStarted","Data":"caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc"}
Nov 01 12:54:40.763376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.763230    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" event={"ID":"4dc78e78-3863-44a7-b0b1-5f96566e8ec8","Type":"ContainerStarted","Data":"7806faa5cb63f2e2982dc80e862137dc5b925df3ab995f0f4dc7115492a4c4a5"}
Nov 01 12:54:40.881627 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:40.881566    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" podStartSLOduration=1.8815495169999998 podStartE2EDuration="1.881549517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:54:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:54:40.88028129 +0000 UTC m=+1940.798076959" watchObservedRunningTime="2026-11-01 12:54:40.881549517 +0000 UTC m=+1940.799345182"
Nov 01 12:54:41.209207 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:41.209176    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:41.214924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:41.214898    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:41.783058 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:41.783022    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:41.783901 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:41.783881    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:54:42.702258 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:42.702220    4095 patch_prober.go:29] interesting pod/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:54:42.702664 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:42.702279    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:54:43.563679 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.563652    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:54:43.676085 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676052    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istiod-ca-cert\") pod \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") "
Nov 01 12:54:43.676262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676145    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-podinfo\") pod \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") "
Nov 01 12:54:43.676262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676167    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-credential-socket\") pod \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") "
Nov 01 12:54:43.676350 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676302    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-envoy\") pod \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") "
Nov 01 12:54:43.676350 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676338    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-socket\") pod \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") "
Nov 01 12:54:43.676438 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676368    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-certs\") pod \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") "
Nov 01 12:54:43.676493 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676448    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-token\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-token\") pod \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") "
Nov 01 12:54:43.676545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676507    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istiod-ca-cert" pod "7809a1c2-11c0-4d8d-90e8-9510a6f3d915" (UID: "7809a1c2-11c0-4d8d-90e8-9510a6f3d915"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:54:43.676545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676533    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-kube-api-access-42hvk\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-kube-api-access-42hvk\") pod \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") "
Nov 01 12:54:43.676649 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676564    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-data\") pod \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\" (UID: \"7809a1c2-11c0-4d8d-90e8-9510a6f3d915\") "
Nov 01 12:54:43.676707 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676622    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-socket" pod "7809a1c2-11c0-4d8d-90e8-9510a6f3d915" (UID: "7809a1c2-11c0-4d8d-90e8-9510a6f3d915"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:43.676707 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676674    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-certs" pod "7809a1c2-11c0-4d8d-90e8-9510a6f3d915" (UID: "7809a1c2-11c0-4d8d-90e8-9510a6f3d915"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:43.676950 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676878    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-credential-socket" pod "7809a1c2-11c0-4d8d-90e8-9510a6f3d915" (UID: "7809a1c2-11c0-4d8d-90e8-9510a6f3d915"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:43.677055 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.676953    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-data" pod "7809a1c2-11c0-4d8d-90e8-9510a6f3d915" (UID: "7809a1c2-11c0-4d8d-90e8-9510a6f3d915"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:43.677377 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.677353    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:43.677468 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.677383    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:43.677468 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.677400    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:43.677468 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.677414    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:43.677468 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.677431    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:43.678552 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.678523    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-podinfo" pod "7809a1c2-11c0-4d8d-90e8-9510a6f3d915" (UID: "7809a1c2-11c0-4d8d-90e8-9510a6f3d915"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:54:43.678645 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.678599    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-envoy" pod "7809a1c2-11c0-4d8d-90e8-9510a6f3d915" (UID: "7809a1c2-11c0-4d8d-90e8-9510a6f3d915"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:54:43.678793 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.678773    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-token" pod "7809a1c2-11c0-4d8d-90e8-9510a6f3d915" (UID: "7809a1c2-11c0-4d8d-90e8-9510a6f3d915"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:43.678839 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.678811    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-kube-api-access-42hvk" pod "7809a1c2-11c0-4d8d-90e8-9510a6f3d915" (UID: "7809a1c2-11c0-4d8d-90e8-9510a6f3d915"). InnerVolumeSpecName "kube-api-access-42hvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:54:43.779529 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.779489    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:43.779529 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.779523    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:43.779529 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.779533    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:43.780015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.779543    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42hvk\" (UniqueName: \"kubernetes.io/projected/7809a1c2-11c0-4d8d-90e8-9510a6f3d915-kube-api-access-42hvk\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:54:43.797362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.797285    4095 generic.go:356] "Generic (PLEG): container finished" podID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerID="48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77" exitCode=0
Nov 01 12:54:43.797362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.797341    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"
Nov 01 12:54:43.797545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.797370    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" event={"ID":"7809a1c2-11c0-4d8d-90e8-9510a6f3d915","Type":"ContainerDied","Data":"48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77"}
Nov 01 12:54:43.797545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.797426    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq" event={"ID":"7809a1c2-11c0-4d8d-90e8-9510a6f3d915","Type":"ContainerDied","Data":"4df176a317208a4a29a9805f28dfd9a404a89b2b298b56f70dd5bb1912b6b731"}
Nov 01 12:54:43.797545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.797449    4095 scope.go:122] "RemoveContainer" containerID="48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77"
Nov 01 12:54:43.808262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.808241    4095 scope.go:122] "RemoveContainer" containerID="48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77"
Nov 01 12:54:43.808527 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:54:43.808509    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77\": container with ID starting with 48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77 not found: ID does not exist" containerID="48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77"
Nov 01 12:54:43.808584 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.808535    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77"} err="failed to get container status \"48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77\": rpc error: code = NotFound desc = could not find container \"48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77\": container with ID starting with 48be214a2f300c9ac3f318afd59c1588b23c708d0c2a841ab39706d5e217aa77 not found: ID does not exist"
Nov 01 12:54:43.964380 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.964346    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"]
Nov 01 12:54:43.974452 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:43.974425    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-ddwbq"]
Nov 01 12:54:44.191559 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:54:44.191528    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" path="/var/lib/kubelet/pods/7809a1c2-11c0-4d8d-90e8-9510a6f3d915/volumes"
Nov 01 12:54:49.116934 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:54:49.116891    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:55:19.116511 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:55:19.116474    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:55:49.116727 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:55:49.116686    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:55:54.860795 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:55:54.860751    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"]
Nov 01 12:55:54.979011 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:55:54.978964    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"]
Nov 01 12:55:59.861312 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:55:59.861247    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4" podUID="427f5e28-9a88-40a4-bb71-4605262b5441" containerName="payload-processing" containerID="cri-o://10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9" gracePeriod=25
Nov 01 12:55:59.979872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:55:59.979829    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l" podUID="96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" containerName="payload-pre-processing" containerID="cri-o://c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776" gracePeriod=25
Nov 01 12:56:01.780289 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:01.778762    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"]
Nov 01 12:56:01.784358 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:01.784323    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw" podUID="4b5bb58d-e902-484d-a456-17c5c88f464f" containerName="maas-api" containerID="cri-o://af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70" gracePeriod=30
Nov 01 12:56:02.127599 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.127569    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:56:02.228604 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.228567    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls\") pod \"4b5bb58d-e902-484d-a456-17c5c88f464f\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") "
Nov 01 12:56:02.228604 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.228609    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-metrics-certs\") pod \"4b5bb58d-e902-484d-a456-17c5c88f464f\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") "
Nov 01 12:56:02.228869 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.228758    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b5bb58d-e902-484d-a456-17c5c88f464f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b5bb58d-e902-484d-a456-17c5c88f464f-openshift-service-ca\") pod \"4b5bb58d-e902-484d-a456-17c5c88f464f\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") "
Nov 01 12:56:02.229099 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.229072    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b5bb58d-e902-484d-a456-17c5c88f464f-kube-api-access-g45p6\" (UniqueName: \"kubernetes.io/projected/4b5bb58d-e902-484d-a456-17c5c88f464f-kube-api-access-g45p6\") pod \"4b5bb58d-e902-484d-a456-17c5c88f464f\" (UID: \"4b5bb58d-e902-484d-a456-17c5c88f464f\") "
Nov 01 12:56:02.229215 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.229159    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b5bb58d-e902-484d-a456-17c5c88f464f-openshift-service-ca" pod "4b5bb58d-e902-484d-a456-17c5c88f464f" (UID: "4b5bb58d-e902-484d-a456-17c5c88f464f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:02.229925 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.229896    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b5bb58d-e902-484d-a456-17c5c88f464f-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:02.231181 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.231152    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-metrics-certs" pod "4b5bb58d-e902-484d-a456-17c5c88f464f" (UID: "4b5bb58d-e902-484d-a456-17c5c88f464f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:56:02.231181 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.231167    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls" pod "4b5bb58d-e902-484d-a456-17c5c88f464f" (UID: "4b5bb58d-e902-484d-a456-17c5c88f464f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:56:02.231305 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.231222    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b5bb58d-e902-484d-a456-17c5c88f464f-kube-api-access-g45p6" pod "4b5bb58d-e902-484d-a456-17c5c88f464f" (UID: "4b5bb58d-e902-484d-a456-17c5c88f464f"). InnerVolumeSpecName "kube-api-access-g45p6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:02.279233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.279130    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l" podUID="96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 12:56:02.279233 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.279168    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4" podUID="427f5e28-9a88-40a4-bb71-4605262b5441" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 12:56:02.331455 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.331418    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:02.331455 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.331450    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4b5bb58d-e902-484d-a456-17c5c88f464f-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:02.331696 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.331465    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g45p6\" (UniqueName: \"kubernetes.io/projected/4b5bb58d-e902-484d-a456-17c5c88f464f-kube-api-access-g45p6\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:02.351388 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.351308    4095 generic.go:356] "Generic (PLEG): container finished" podID="4b5bb58d-e902-484d-a456-17c5c88f464f" containerID="af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70" exitCode=0
Nov 01 12:56:02.351388 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.351346    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw" event={"ID":"4b5bb58d-e902-484d-a456-17c5c88f464f","Type":"ContainerDied","Data":"af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70"}
Nov 01 12:56:02.351605 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.351406    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw" event={"ID":"4b5bb58d-e902-484d-a456-17c5c88f464f","Type":"ContainerDied","Data":"3ba7fe041e840de1f9ceab4609bb6bd4f31f76891d0cf01f51d13ccbb11f7c14"}
Nov 01 12:56:02.351605 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.351428    4095 scope.go:122] "RemoveContainer" containerID="af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70"
Nov 01 12:56:02.351605 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.351354    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"
Nov 01 12:56:02.376003 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.372855    4095 scope.go:122] "RemoveContainer" containerID="af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70"
Nov 01 12:56:02.376395 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:02.376368    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70\": container with ID starting with af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70 not found: ID does not exist" containerID="af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70"
Nov 01 12:56:02.376507 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.376404    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70"} err="failed to get container status \"af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70\": rpc error: code = NotFound desc = could not find container \"af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70\": container with ID starting with af643f7dd9aa7983b807c0954ca179d3742eba6002113761c0c2eb5afe75cf70 not found: ID does not exist"
Nov 01 12:56:02.516585 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.516552    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"]
Nov 01 12:56:02.519690 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:02.519666    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-f6b043-7f67646b49-q6hzw"]
Nov 01 12:56:04.189988 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:04.189937    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b5bb58d-e902-484d-a456-17c5c88f464f" path="/var/lib/kubelet/pods/4b5bb58d-e902-484d-a456-17c5c88f464f/volumes"
Nov 01 12:56:11.137801 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:11.137754    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"]
Nov 01 12:56:11.141692 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:11.138371    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy" containerID="cri-o://fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d" gracePeriod=30
Nov 01 12:56:12.276647 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:12.276613    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l" podUID="96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 12:56:12.277073 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:12.276656    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4" podUID="427f5e28-9a88-40a4-bb71-4605262b5441" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 12:56:16.440491 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.440462    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:56:16.451472 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.451442    4095 generic.go:356] "Generic (PLEG): container finished" podID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerID="fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d" exitCode=0
Nov 01 12:56:16.451656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.451512    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"
Nov 01 12:56:16.451656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.451517    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" event={"ID":"6b0c3640-96a4-4192-adf9-adc86d71d430","Type":"ContainerDied","Data":"fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d"}
Nov 01 12:56:16.451656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.451591    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj" event={"ID":"6b0c3640-96a4-4192-adf9-adc86d71d430","Type":"ContainerDied","Data":"ef170580196abac26404e2874fdc4a171ab55b10a9b354b8f528918994b069e6"}
Nov 01 12:56:16.451656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.451613    4095 scope.go:122] "RemoveContainer" containerID="fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d"
Nov 01 12:56:16.461762 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.461744    4095 scope.go:122] "RemoveContainer" containerID="25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3"
Nov 01 12:56:16.471110 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.471087    4095 scope.go:122] "RemoveContainer" containerID="fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d"
Nov 01 12:56:16.471402 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:16.471382    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d\": container with ID starting with fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d not found: ID does not exist" containerID="fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d"
Nov 01 12:56:16.471485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.471415    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d"} err="failed to get container status \"fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d\": rpc error: code = NotFound desc = could not find container \"fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d\": container with ID starting with fd016f3f2885df1651538a0fc32df459608892b415fa0c2344400ffbe065c17d not found: ID does not exist"
Nov 01 12:56:16.471485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.471442    4095 scope.go:122] "RemoveContainer" containerID="25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3"
Nov 01 12:56:16.471683 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:16.471661    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3\": container with ID starting with 25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3 not found: ID does not exist" containerID="25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3"
Nov 01 12:56:16.471730 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.471689    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3"} err="failed to get container status \"25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3\": rpc error: code = NotFound desc = could not find container \"25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3\": container with ID starting with 25f899c690e40e97c4dbb72ee6701a7c0ede121039036b5e37a4ff40d4c444f3 not found: ID does not exist"
Nov 01 12:56:16.605958 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.605922    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-envoy\") pod \"6b0c3640-96a4-4192-adf9-adc86d71d430\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") "
Nov 01 12:56:16.606165 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.605965    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-podinfo\") pod \"6b0c3640-96a4-4192-adf9-adc86d71d430\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") "
Nov 01 12:56:16.606165 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.606022    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-socket\") pod \"6b0c3640-96a4-4192-adf9-adc86d71d430\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") "
Nov 01 12:56:16.606165 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.606051    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-credential-socket\") pod \"6b0c3640-96a4-4192-adf9-adc86d71d430\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") "
Nov 01 12:56:16.606332 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.606165    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-data\") pod \"6b0c3640-96a4-4192-adf9-adc86d71d430\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") "
Nov 01 12:56:16.606332 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.606281    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-kube-api-access-wms9h\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-kube-api-access-wms9h\") pod \"6b0c3640-96a4-4192-adf9-adc86d71d430\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") "
Nov 01 12:56:16.606332 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.606315    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b0c3640-96a4-4192-adf9-adc86d71d430-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c3640-96a4-4192-adf9-adc86d71d430-istiod-ca-cert\") pod \"6b0c3640-96a4-4192-adf9-adc86d71d430\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") "
Nov 01 12:56:16.606481 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.606390    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-token\") pod \"6b0c3640-96a4-4192-adf9-adc86d71d430\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") "
Nov 01 12:56:16.606481 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.606467    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-certs\") pod \"6b0c3640-96a4-4192-adf9-adc86d71d430\" (UID: \"6b0c3640-96a4-4192-adf9-adc86d71d430\") "
Nov 01 12:56:16.606839 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.606809    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b0c3640-96a4-4192-adf9-adc86d71d430-istiod-ca-cert" pod "6b0c3640-96a4-4192-adf9-adc86d71d430" (UID: "6b0c3640-96a4-4192-adf9-adc86d71d430"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:16.606959 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.606874    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-data" pod "6b0c3640-96a4-4192-adf9-adc86d71d430" (UID: "6b0c3640-96a4-4192-adf9-adc86d71d430"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:16.607105 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.607080    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-certs" pod "6b0c3640-96a4-4192-adf9-adc86d71d430" (UID: "6b0c3640-96a4-4192-adf9-adc86d71d430"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:16.607362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.607332    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:16.607458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.607362    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b0c3640-96a4-4192-adf9-adc86d71d430-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:16.607458 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.607378    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:16.607574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.607457    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-credential-socket" pod "6b0c3640-96a4-4192-adf9-adc86d71d430" (UID: "6b0c3640-96a4-4192-adf9-adc86d71d430"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:16.607815 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.607784    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-socket" pod "6b0c3640-96a4-4192-adf9-adc86d71d430" (UID: "6b0c3640-96a4-4192-adf9-adc86d71d430"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:16.608763 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.608732    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-envoy" pod "6b0c3640-96a4-4192-adf9-adc86d71d430" (UID: "6b0c3640-96a4-4192-adf9-adc86d71d430"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:16.609175 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.609152    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-kube-api-access-wms9h" pod "6b0c3640-96a4-4192-adf9-adc86d71d430" (UID: "6b0c3640-96a4-4192-adf9-adc86d71d430"). InnerVolumeSpecName "kube-api-access-wms9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:16.609307 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.609293    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-podinfo" pod "6b0c3640-96a4-4192-adf9-adc86d71d430" (UID: "6b0c3640-96a4-4192-adf9-adc86d71d430"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:56:16.609634 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.609612    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-token" pod "6b0c3640-96a4-4192-adf9-adc86d71d430" (UID: "6b0c3640-96a4-4192-adf9-adc86d71d430"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:16.708625 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.708547    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:16.708625 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.708575    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:16.708625 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.708585    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:16.708625 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.708593    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b0c3640-96a4-4192-adf9-adc86d71d430-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:16.708625 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.708606    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wms9h\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-kube-api-access-wms9h\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:16.708625 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.708620    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b0c3640-96a4-4192-adf9-adc86d71d430-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:16.868242 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.868201    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"]
Nov 01 12:56:16.871460 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:16.871437    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-f6b043-openshift-default-5f6df66cfc-pt5pj"]
Nov 01 12:56:18.191655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:18.191624    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" path="/var/lib/kubelet/pods/6b0c3640-96a4-4192-adf9-adc86d71d430/volumes"
Nov 01 12:56:19.116777 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:19.116745    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:56:20.181318 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.181288    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:56:20.270448 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.270420    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:56:20.364496 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.364465    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-kube-api-access-bxrq8\" (UniqueName: \"kubernetes.io/projected/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-kube-api-access-bxrq8\") pod \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\" (UID: \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\") "
Nov 01 12:56:20.364674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.364512    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/427f5e28-9a88-40a4-bb71-4605262b5441-kube-api-access-dkgm5\" (UniqueName: \"kubernetes.io/projected/427f5e28-9a88-40a4-bb71-4605262b5441-kube-api-access-dkgm5\") pod \"427f5e28-9a88-40a4-bb71-4605262b5441\" (UID: \"427f5e28-9a88-40a4-bb71-4605262b5441\") "
Nov 01 12:56:20.364674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.364601    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-plugins-config-volume\") pod \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\" (UID: \"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4\") "
Nov 01 12:56:20.364674 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.364646    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/427f5e28-9a88-40a4-bb71-4605262b5441-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427f5e28-9a88-40a4-bb71-4605262b5441-plugins-config-volume\") pod \"427f5e28-9a88-40a4-bb71-4605262b5441\" (UID: \"427f5e28-9a88-40a4-bb71-4605262b5441\") "
Nov 01 12:56:20.365075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.365044    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-plugins-config-volume" pod "96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" (UID: "96f6212b-9ad5-4da9-b4ce-1ae647bdbff4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:20.365194 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.365086    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/427f5e28-9a88-40a4-bb71-4605262b5441-plugins-config-volume" pod "427f5e28-9a88-40a4-bb71-4605262b5441" (UID: "427f5e28-9a88-40a4-bb71-4605262b5441"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:20.365302 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.365286    4095 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-plugins-config-volume\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:20.365344 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.365305    4095 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/427f5e28-9a88-40a4-bb71-4605262b5441-plugins-config-volume\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:20.366656 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.366635    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-kube-api-access-bxrq8" pod "96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" (UID: "96f6212b-9ad5-4da9-b4ce-1ae647bdbff4"). InnerVolumeSpecName "kube-api-access-bxrq8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:20.366788 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.366771    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/427f5e28-9a88-40a4-bb71-4605262b5441-kube-api-access-dkgm5" pod "427f5e28-9a88-40a4-bb71-4605262b5441" (UID: "427f5e28-9a88-40a4-bb71-4605262b5441"). InnerVolumeSpecName "kube-api-access-dkgm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:20.466639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.466569    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxrq8\" (UniqueName: \"kubernetes.io/projected/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4-kube-api-access-bxrq8\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:20.466639 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.466594    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkgm5\" (UniqueName: \"kubernetes.io/projected/427f5e28-9a88-40a4-bb71-4605262b5441-kube-api-access-dkgm5\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:20.488889 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.488847    4095 generic.go:356] "Generic (PLEG): container finished" podID="427f5e28-9a88-40a4-bb71-4605262b5441" containerID="10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9" exitCode=0
Nov 01 12:56:20.489090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.488903    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"
Nov 01 12:56:20.489090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.488932    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4" event={"ID":"427f5e28-9a88-40a4-bb71-4605262b5441","Type":"ContainerDied","Data":"10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9"}
Nov 01 12:56:20.489090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.489035    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4" event={"ID":"427f5e28-9a88-40a4-bb71-4605262b5441","Type":"ContainerDied","Data":"69d17730f410af369eada98cfe5e922dc6b4b8f32ccaede30ec0525e86586946"}
Nov 01 12:56:20.489090 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.489066    4095 scope.go:122] "RemoveContainer" containerID="10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9"
Nov 01 12:56:20.491039 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.491008    4095 generic.go:356] "Generic (PLEG): container finished" podID="96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" containerID="c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776" exitCode=0
Nov 01 12:56:20.491171 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.491067    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"
Nov 01 12:56:20.491171 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.491096    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l" event={"ID":"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4","Type":"ContainerDied","Data":"c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776"}
Nov 01 12:56:20.491171 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.491137    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l" event={"ID":"96f6212b-9ad5-4da9-b4ce-1ae647bdbff4","Type":"ContainerDied","Data":"5adfecf8a47b3761d1400bce440133fc060bc4e4adbe808d066209b632de2758"}
Nov 01 12:56:20.501824 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.501782    4095 scope.go:122] "RemoveContainer" containerID="10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9"
Nov 01 12:56:20.502312 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:20.502285    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9\": container with ID starting with 10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9 not found: ID does not exist" containerID="10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9"
Nov 01 12:56:20.502401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.502324    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9"} err="failed to get container status \"10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9\": rpc error: code = NotFound desc = could not find container \"10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9\": container with ID starting with 10f73c4a0e90eb5adccf7473df449da01582b31759a5ad901d7785e9308fd8e9 not found: ID does not exist"
Nov 01 12:56:20.502401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.502348    4095 scope.go:122] "RemoveContainer" containerID="c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776"
Nov 01 12:56:20.512551 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.512531    4095 scope.go:122] "RemoveContainer" containerID="c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776"
Nov 01 12:56:20.512870 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:20.512842    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776\": container with ID starting with c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776 not found: ID does not exist" containerID="c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776"
Nov 01 12:56:20.513037 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.512879    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776"} err="failed to get container status \"c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776\": rpc error: code = NotFound desc = could not find container \"c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776\": container with ID starting with c17019b6c6d251c8901e4076eef2ee079a6ebc33c8f19a767eb39827b28a3776 not found: ID does not exist"
Nov 01 12:56:20.648485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.648459    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"]
Nov 01 12:56:20.651527 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.651507    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-187d06ed-6cd8c49b6c-mz79l"]
Nov 01 12:56:20.731113 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731089    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"]
Nov 01 12:56:20.731728 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731713    4095 cpu_manager.go:414] "containerMap: removing container" podUID="4b5bb58d-e902-484d-a456-17c5c88f464f" containerName="maas-api"
Nov 01 12:56:20.731769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731731    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b5bb58d-e902-484d-a456-17c5c88f464f" containerName="maas-api"
Nov 01 12:56:20.731769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731745    4095 cpu_manager.go:414] "containerMap: removing container" podUID="427f5e28-9a88-40a4-bb71-4605262b5441" containerName="payload-processing"
Nov 01 12:56:20.731769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731751    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="427f5e28-9a88-40a4-bb71-4605262b5441" containerName="payload-processing"
Nov 01 12:56:20.731769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731762    4095 cpu_manager.go:414] "containerMap: removing container" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy"
Nov 01 12:56:20.731769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731767    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy"
Nov 01 12:56:20.731918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731775    4095 cpu_manager.go:414] "containerMap: removing container" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy"
Nov 01 12:56:20.731918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731781    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy"
Nov 01 12:56:20.731918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731795    4095 cpu_manager.go:414] "containerMap: removing container" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy"
Nov 01 12:56:20.731918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731801    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy"
Nov 01 12:56:20.731918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731812    4095 cpu_manager.go:414] "containerMap: removing container" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy"
Nov 01 12:56:20.731918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731817    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy"
Nov 01 12:56:20.731918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731837    4095 cpu_manager.go:414] "containerMap: removing container" podUID="96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" containerName="payload-pre-processing"
Nov 01 12:56:20.731918 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731841    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" containerName="payload-pre-processing"
Nov 01 12:56:20.732158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731934    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" containerName="payload-pre-processing"
Nov 01 12:56:20.732158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731950    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b5bb58d-e902-484d-a456-17c5c88f464f" containerName="maas-api"
Nov 01 12:56:20.732158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731955    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="427f5e28-9a88-40a4-bb71-4605262b5441" containerName="payload-processing"
Nov 01 12:56:20.732158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731962    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="7809a1c2-11c0-4d8d-90e8-9510a6f3d915" containerName="istio-proxy"
Nov 01 12:56:20.732158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731981    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy"
Nov 01 12:56:20.732158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731989    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2d01769-afe3-402d-999b-0ecc4a64814f" containerName="istio-proxy"
Nov 01 12:56:20.732158 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.731997    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b0c3640-96a4-4192-adf9-adc86d71d430" containerName="istio-proxy"
Nov 01 12:56:20.737127 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.737103    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.757014 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.756988    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"]
Nov 01 12:56:20.760102 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.760079    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-187d06ed-d7db5c597-q97r4"]
Nov 01 12:56:20.769456 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.769433    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.769570 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.769467    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.769644 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.769630    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn582\" (UniqueName: \"kubernetes.io/projected/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-kube-api-access-qn582\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.769716 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.769660    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.789499 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.789468    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-78c69d-serving-cert\""
Nov 01 12:56:20.790505 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.790485    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"]
Nov 01 12:56:20.870937 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.870897    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn582\" (UniqueName: \"kubernetes.io/projected/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-kube-api-access-qn582\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.871121 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.870964    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.871121 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:20.871081    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-serving-cert: secret "maas-api-e2e-shared-b-w4-78c69d-serving-cert" not found
Nov 01 12:56:20.871242 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:20.871149    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls podName:fff1b7f2-ba38-4aab-94f8-00c8f23ef154 nodeName:}" failed. No retries permitted until 2026-11-01 12:56:21.371133573 +0000 UTC m=+2041.288929218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls") pod "maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54" (UID: "fff1b7f2-ba38-4aab-94f8-00c8f23ef154") : secret "maas-api-e2e-shared-b-w4-78c69d-serving-cert" not found
Nov 01 12:56:20.871242 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.871189    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.871242 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.871215    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.872030 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.872009    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.873655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.873634    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:20.928032 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:20.928006    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn582\" (UniqueName: \"kubernetes.io/projected/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-kube-api-access-qn582\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:21.380427 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:21.380388    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:21.383032 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:21.383008    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:21.652123 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:21.652048    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:22.023330 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:22.023290    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"]
Nov 01 12:56:22.023478 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:56:22.023453    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff1b7f2_ba38_4aab_94f8_00c8f23ef154.slice/crio-6464c9c32120174a2e8b502f63791e19530fc045572559fa6e02f831365b3860 WatchSource:0}: Error finding container 6464c9c32120174a2e8b502f63791e19530fc045572559fa6e02f831365b3860: Status 404 returned error can't find the container with id 6464c9c32120174a2e8b502f63791e19530fc045572559fa6e02f831365b3860
Nov 01 12:56:22.025080 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:22.025060    4095 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 12:56:22.191313 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:22.191285    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="427f5e28-9a88-40a4-bb71-4605262b5441" path="/var/lib/kubelet/pods/427f5e28-9a88-40a4-bb71-4605262b5441/volumes"
Nov 01 12:56:22.191731 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:22.191718    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96f6212b-9ad5-4da9-b4ce-1ae647bdbff4" path="/var/lib/kubelet/pods/96f6212b-9ad5-4da9-b4ce-1ae647bdbff4/volumes"
Nov 01 12:56:22.505790 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:22.505755    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54" event={"ID":"fff1b7f2-ba38-4aab-94f8-00c8f23ef154","Type":"ContainerStarted","Data":"6464c9c32120174a2e8b502f63791e19530fc045572559fa6e02f831365b3860"}
Nov 01 12:56:23.515741 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:23.515698    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54" event={"ID":"fff1b7f2-ba38-4aab-94f8-00c8f23ef154","Type":"ContainerStarted","Data":"ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb"}
Nov 01 12:56:23.516187 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:23.515927    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:23.630029 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:23.629942    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54" podStartSLOduration=2.834042236 podStartE2EDuration="3.62992369s" totalImagesPullingTime="795.881454ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:56:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:56:23.625313845 +0000 UTC m=+2043.543109514" watchObservedRunningTime="2026-11-01 12:56:23.62992369 +0000 UTC m=+2043.547719359"
Nov 01 12:56:29.543840 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:29.543811    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 12:56:47.702889 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:47.702857    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"]
Nov 01 12:56:47.703339 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:47.703065    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy" containerID="cri-o://15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c" gracePeriod=30
Nov 01 12:56:47.760089 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:47.760059    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:56:47.760260 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:47.760115    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:56:49.116052 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:49.116011    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:56:52.980729 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:52.980669    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:56:53.082767 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.082736    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-envoy\") pod \"1064c601-a2a0-42af-9fcc-5886f08617e4\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") "
Nov 01 12:56:53.082953 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.082833    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-certs\") pod \"1064c601-a2a0-42af-9fcc-5886f08617e4\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") "
Nov 01 12:56:53.082953 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.082879    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1064c601-a2a0-42af-9fcc-5886f08617e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1064c601-a2a0-42af-9fcc-5886f08617e4-istiod-ca-cert\") pod \"1064c601-a2a0-42af-9fcc-5886f08617e4\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") "
Nov 01 12:56:53.082953 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.082939    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-credential-socket\") pod \"1064c601-a2a0-42af-9fcc-5886f08617e4\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") "
Nov 01 12:56:53.083155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.083067    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-token\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-token\") pod \"1064c601-a2a0-42af-9fcc-5886f08617e4\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") "
Nov 01 12:56:53.083155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.083100    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-kube-api-access-lgvrr\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-kube-api-access-lgvrr\") pod \"1064c601-a2a0-42af-9fcc-5886f08617e4\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") "
Nov 01 12:56:53.083155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.083142    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-certs" pod "1064c601-a2a0-42af-9fcc-5886f08617e4" (UID: "1064c601-a2a0-42af-9fcc-5886f08617e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:53.083323 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.083162    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-data\") pod \"1064c601-a2a0-42af-9fcc-5886f08617e4\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") "
Nov 01 12:56:53.083323 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.083207    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-socket\") pod \"1064c601-a2a0-42af-9fcc-5886f08617e4\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") "
Nov 01 12:56:53.083323 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.083277    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-podinfo\") pod \"1064c601-a2a0-42af-9fcc-5886f08617e4\" (UID: \"1064c601-a2a0-42af-9fcc-5886f08617e4\") "
Nov 01 12:56:53.083487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.083406    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-credential-socket" pod "1064c601-a2a0-42af-9fcc-5886f08617e4" (UID: "1064c601-a2a0-42af-9fcc-5886f08617e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:53.083487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.083424    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-data" pod "1064c601-a2a0-42af-9fcc-5886f08617e4" (UID: "1064c601-a2a0-42af-9fcc-5886f08617e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:53.083626 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.083601    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-socket" pod "1064c601-a2a0-42af-9fcc-5886f08617e4" (UID: "1064c601-a2a0-42af-9fcc-5886f08617e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:53.084046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.084023    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:53.084157 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.084048    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:53.084157 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.084063    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:53.084157 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.084077    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:53.084157 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.084115    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1064c601-a2a0-42af-9fcc-5886f08617e4-istiod-ca-cert" pod "1064c601-a2a0-42af-9fcc-5886f08617e4" (UID: "1064c601-a2a0-42af-9fcc-5886f08617e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:56:53.085173 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.085146    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-envoy" pod "1064c601-a2a0-42af-9fcc-5886f08617e4" (UID: "1064c601-a2a0-42af-9fcc-5886f08617e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:56:53.085293 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.085270    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-token" pod "1064c601-a2a0-42af-9fcc-5886f08617e4" (UID: "1064c601-a2a0-42af-9fcc-5886f08617e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:53.085339 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.085294    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-kube-api-access-lgvrr" pod "1064c601-a2a0-42af-9fcc-5886f08617e4" (UID: "1064c601-a2a0-42af-9fcc-5886f08617e4"). InnerVolumeSpecName "kube-api-access-lgvrr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:56:53.085498 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.085483    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-podinfo" pod "1064c601-a2a0-42af-9fcc-5886f08617e4" (UID: "1064c601-a2a0-42af-9fcc-5886f08617e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:56:53.185125 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.185093    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:53.185125 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.185126    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgvrr\" (UniqueName: \"kubernetes.io/projected/1064c601-a2a0-42af-9fcc-5886f08617e4-kube-api-access-lgvrr\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:53.185125 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.185139    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:53.185417 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.185151    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1064c601-a2a0-42af-9fcc-5886f08617e4-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:53.185417 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.185163    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1064c601-a2a0-42af-9fcc-5886f08617e4-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:56:53.313817 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.313734    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"]
Nov 01 12:56:53.322907 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.322878    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"]
Nov 01 12:56:53.333028 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.332998    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"]
Nov 01 12:56:53.734399 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.734368    4095 generic.go:356] "Generic (PLEG): container finished" podID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerID="15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c" exitCode=0
Nov 01 12:56:53.734603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.734422    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"
Nov 01 12:56:53.734603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.734445    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" event={"ID":"1064c601-a2a0-42af-9fcc-5886f08617e4","Type":"ContainerDied","Data":"15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c"}
Nov 01 12:56:53.734603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.734509    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj" event={"ID":"1064c601-a2a0-42af-9fcc-5886f08617e4","Type":"ContainerDied","Data":"34e8552d04f3c0d464f91655730c52f4170d15059238b77c63ff4832cd67700b"}
Nov 01 12:56:53.734603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.734539    4095 scope.go:122] "RemoveContainer" containerID="15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c"
Nov 01 12:56:53.745334 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.745316    4095 scope.go:122] "RemoveContainer" containerID="e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620"
Nov 01 12:56:53.754624 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.754597    4095 scope.go:122] "RemoveContainer" containerID="15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c"
Nov 01 12:56:53.754911 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:53.754884    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c\": container with ID starting with 15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c not found: ID does not exist" containerID="15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c"
Nov 01 12:56:53.755034 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.754927    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c"} err="failed to get container status \"15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c\": rpc error: code = NotFound desc = could not find container \"15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c\": container with ID starting with 15e75baf3a64476789dcf4779da762dc02cc15610e02254ad28317f12ed4542c not found: ID does not exist"
Nov 01 12:56:53.755034 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.754952    4095 scope.go:122] "RemoveContainer" containerID="e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620"
Nov 01 12:56:53.755243 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:56:53.755224    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620\": container with ID starting with e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620 not found: ID does not exist" containerID="e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620"
Nov 01 12:56:53.755285 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.755250    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620"} err="failed to get container status \"e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620\": rpc error: code = NotFound desc = could not find container \"e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620\": container with ID starting with e765d3bf895659b00a71a89d5ca2b543bd390b4f403a1a641b801c1f7222d620 not found: ID does not exist"
Nov 01 12:56:53.892909 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.892877    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"]
Nov 01 12:56:53.897141 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:53.897116    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-c4bbec-openshift-default-75f4b96ccf-hjkjj"]
Nov 01 12:56:54.191218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:56:54.191184    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" path="/var/lib/kubelet/pods/1064c601-a2a0-42af-9fcc-5886f08617e4/volumes"
Nov 01 12:57:06.925545 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:06.925509    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"]
Nov 01 12:57:06.926360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:06.926322    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy" containerID="cri-o://850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970" gracePeriod=30
Nov 01 12:57:06.960345 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:06.960312    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"]
Nov 01 12:57:06.960550 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:06.960525    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy" containerID="cri-o://8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6" gracePeriod=30
Nov 01 12:57:07.167106 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:07.167069    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"]
Nov 01 12:57:07.167336 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:07.167313    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy" containerID="cri-o://162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c" gracePeriod=30
Nov 01 12:57:07.170568 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:07.170533    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"]
Nov 01 12:57:07.170790 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:07.170754    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy" containerID="cri-o://21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc" gracePeriod=30
Nov 01 12:57:08.058091 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.058052    4095 patch_prober.go:29] interesting pod/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:57:08.058488 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.058123    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:57:08.335795 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.335674    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" containerID="cri-o://2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0" gracePeriod=45
Nov 01 12:57:08.344114 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.344057    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" containerID="cri-o://37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243" gracePeriod=45
Nov 01 12:57:08.353502 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.353474    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main" containerID="cri-o://4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490" gracePeriod=45
Nov 01 12:57:08.514448 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.514140    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": dial tcp 10.134.0.56:8000: connect: connection refused"
Nov 01 12:57:08.682191 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.681795    4095 patch_prober.go:29] interesting pod/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:57:08.682191 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.681854    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:57:08.696483 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.696458    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:57:08.734653 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.734518    4095 patch_prober.go:29] interesting pod/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:57:08.734653 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.734582    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:57:08.745868 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.745839    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:57:08.763636 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.763614    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:57:08.842453 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.841406    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tls-certs\") pod \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") "
Nov 01 12:57:08.842671 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.842516    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-home\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-home\") pod \"54698a98-165c-4f90-a571-6fee68c2d0f1\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") "
Nov 01 12:57:08.842872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.842784    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-home" pod "54698a98-165c-4f90-a571-6fee68c2d0f1" (UID: "54698a98-165c-4f90-a571-6fee68c2d0f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.842872 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.842854    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-model-cache\") pod \"54698a98-165c-4f90-a571-6fee68c2d0f1\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") "
Nov 01 12:57:08.843134 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.843115    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-model-cache" pod "54698a98-165c-4f90-a571-6fee68c2d0f1" (UID: "54698a98-165c-4f90-a571-6fee68c2d0f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.843475 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.843448    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/54698a98-165c-4f90-a571-6fee68c2d0f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/54698a98-165c-4f90-a571-6fee68c2d0f1-tls-certs\") pod \"54698a98-165c-4f90-a571-6fee68c2d0f1\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") "
Nov 01 12:57:08.843570 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.843518    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-tmp-dir\") pod \"54698a98-165c-4f90-a571-6fee68c2d0f1\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") "
Nov 01 12:57:08.843651 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.843634    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-kube-api-access-lgcs5\" (UniqueName: \"kubernetes.io/projected/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-kube-api-access-lgcs5\") pod \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") "
Nov 01 12:57:08.844210 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.843932    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-tmp-dir" pod "54698a98-165c-4f90-a571-6fee68c2d0f1" (UID: "54698a98-165c-4f90-a571-6fee68c2d0f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.844210 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.844118    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-model-cache\") pod \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") "
Nov 01 12:57:08.844210 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.844151    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-dshm\") pod \"54698a98-165c-4f90-a571-6fee68c2d0f1\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") "
Nov 01 12:57:08.844210 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.844170    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-model-cache" pod "3e8cde83-0122-42ac-a2d5-e6c2e6d97703" (UID: "3e8cde83-0122-42ac-a2d5-e6c2e6d97703"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.844472 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.844235    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-dshm\") pod \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") "
Nov 01 12:57:08.844472 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.844347    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tls-certs" pod "3e8cde83-0122-42ac-a2d5-e6c2e6d97703" (UID: "3e8cde83-0122-42ac-a2d5-e6c2e6d97703"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:08.844472 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.844369    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54698a98-165c-4f90-a571-6fee68c2d0f1-kube-api-access-456fh\" (UniqueName: \"kubernetes.io/projected/54698a98-165c-4f90-a571-6fee68c2d0f1-kube-api-access-456fh\") pod \"54698a98-165c-4f90-a571-6fee68c2d0f1\" (UID: \"54698a98-165c-4f90-a571-6fee68c2d0f1\") "
Nov 01 12:57:08.844719 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.844701    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tmp-dir\") pod \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") "
Nov 01 12:57:08.845074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.844795    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-home\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-home\") pod \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\" (UID: \"3e8cde83-0122-42ac-a2d5-e6c2e6d97703\") "
Nov 01 12:57:08.845074 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.845034    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tmp-dir" pod "3e8cde83-0122-42ac-a2d5-e6c2e6d97703" (UID: "3e8cde83-0122-42ac-a2d5-e6c2e6d97703"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.845401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.845368    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-home" pod "3e8cde83-0122-42ac-a2d5-e6c2e6d97703" (UID: "3e8cde83-0122-42ac-a2d5-e6c2e6d97703"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.845401 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.845384    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/54698a98-165c-4f90-a571-6fee68c2d0f1-tls-certs" pod "54698a98-165c-4f90-a571-6fee68c2d0f1" (UID: "54698a98-165c-4f90-a571-6fee68c2d0f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:08.846046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846023    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.846142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846050    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.846142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846063    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.846142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846076    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.846142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846091    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.846142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846108    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54698a98-165c-4f90-a571-6fee68c2d0f1-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.846142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846123    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.846142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846136    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.846687 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846664    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-kube-api-access-lgcs5" pod "3e8cde83-0122-42ac-a2d5-e6c2e6d97703" (UID: "3e8cde83-0122-42ac-a2d5-e6c2e6d97703"). InnerVolumeSpecName "kube-api-access-lgcs5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:08.846809 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.846766    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-dshm" pod "54698a98-165c-4f90-a571-6fee68c2d0f1" (UID: "54698a98-165c-4f90-a571-6fee68c2d0f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.847777 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.847757    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-dshm" pod "3e8cde83-0122-42ac-a2d5-e6c2e6d97703" (UID: "3e8cde83-0122-42ac-a2d5-e6c2e6d97703"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.848364 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.848334    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54698a98-165c-4f90-a571-6fee68c2d0f1-kube-api-access-456fh" pod "54698a98-165c-4f90-a571-6fee68c2d0f1" (UID: "54698a98-165c-4f90-a571-6fee68c2d0f1"). InnerVolumeSpecName "kube-api-access-456fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:08.863934 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.863902    4095 generic.go:356] "Generic (PLEG): container finished" podID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerID="37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243" exitCode=0
Nov 01 12:57:08.864194 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.863958    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"
Nov 01 12:57:08.864194 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.863993    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" event={"ID":"54698a98-165c-4f90-a571-6fee68c2d0f1","Type":"ContainerDied","Data":"37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243"}
Nov 01 12:57:08.864194 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.864042    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" event={"ID":"54698a98-165c-4f90-a571-6fee68c2d0f1","Type":"ContainerDied","Data":"c1df9a5a077b164d473793824805fb0b474f4c9bc7e733769c7f33bca50e4afd"}
Nov 01 12:57:08.864194 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.864067    4095 scope.go:122] "RemoveContainer" containerID="37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243"
Nov 01 12:57:08.866046 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.866025    4095 generic.go:356] "Generic (PLEG): container finished" podID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerID="2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0" exitCode=0
Nov 01 12:57:08.866199 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.866111    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" event={"ID":"3e8cde83-0122-42ac-a2d5-e6c2e6d97703","Type":"ContainerDied","Data":"2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0"}
Nov 01 12:57:08.866199 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.866127    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"
Nov 01 12:57:08.866199 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.866143    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv" event={"ID":"3e8cde83-0122-42ac-a2d5-e6c2e6d97703","Type":"ContainerDied","Data":"e8a8922c2c183f76a64963114eb37833442c381ac18631cc051af08d1945f397"}
Nov 01 12:57:08.870178 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.869954    4095 generic.go:356] "Generic (PLEG): container finished" podID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerID="4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490" exitCode=0
Nov 01 12:57:08.870178 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.870083    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"
Nov 01 12:57:08.870178 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.870130    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" event={"ID":"a290f39b-7f4c-4927-8fa3-3094bb02374c","Type":"ContainerDied","Data":"4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490"}
Nov 01 12:57:08.870178 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.870157    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s" event={"ID":"a290f39b-7f4c-4927-8fa3-3094bb02374c","Type":"ContainerDied","Data":"1941f5922c319750128627fc576b25943d853917ac94ebd0d3c76836f55378d9"}
Nov 01 12:57:08.889220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.880204    4095 scope.go:122] "RemoveContainer" containerID="37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243"
Nov 01 12:57:08.889220 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:08.880605    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243\": container with ID starting with 37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243 not found: ID does not exist" containerID="37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243"
Nov 01 12:57:08.889220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.880640    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243"} err="failed to get container status \"37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243\": rpc error: code = NotFound desc = could not find container \"37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243\": container with ID starting with 37cb3f40d37beb8288aa350e44775d6f991d3917124d287aae0e9834e4bd1243 not found: ID does not exist"
Nov 01 12:57:08.889220 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.880663    4095 scope.go:122] "RemoveContainer" containerID="2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0"
Nov 01 12:57:08.900159 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.900139    4095 scope.go:122] "RemoveContainer" containerID="2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0"
Nov 01 12:57:08.900725 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:08.900575    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0\": container with ID starting with 2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0 not found: ID does not exist" containerID="2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0"
Nov 01 12:57:08.900725 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.900612    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0"} err="failed to get container status \"2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0\": rpc error: code = NotFound desc = could not find container \"2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0\": container with ID starting with 2f779eb054d9ae93112b79599e45be5f039b90a66c24aa461f2c12697a8dbdf0 not found: ID does not exist"
Nov 01 12:57:08.900725 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.900637    4095 scope.go:122] "RemoveContainer" containerID="4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490"
Nov 01 12:57:08.949449 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.949422    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-dshm\") pod \"a290f39b-7f4c-4927-8fa3-3094bb02374c\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") "
Nov 01 12:57:08.949659 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.949646    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a290f39b-7f4c-4927-8fa3-3094bb02374c-tls-certs\" (UniqueName: \"kubernetes.io/secret/a290f39b-7f4c-4927-8fa3-3094bb02374c-tls-certs\") pod \"a290f39b-7f4c-4927-8fa3-3094bb02374c\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") "
Nov 01 12:57:08.949834 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.949822    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-model-cache\") pod \"a290f39b-7f4c-4927-8fa3-3094bb02374c\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") "
Nov 01 12:57:08.949925 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.949915    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a290f39b-7f4c-4927-8fa3-3094bb02374c-kube-api-access-z8vfg\" (UniqueName: \"kubernetes.io/projected/a290f39b-7f4c-4927-8fa3-3094bb02374c-kube-api-access-z8vfg\") pod \"a290f39b-7f4c-4927-8fa3-3094bb02374c\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") "
Nov 01 12:57:08.950332 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.950313    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-home\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-home\") pod \"a290f39b-7f4c-4927-8fa3-3094bb02374c\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") "
Nov 01 12:57:08.950476 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.950464    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-tmp-dir\") pod \"a290f39b-7f4c-4927-8fa3-3094bb02374c\" (UID: \"a290f39b-7f4c-4927-8fa3-3094bb02374c\") "
Nov 01 12:57:08.950845 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.950735    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-home" pod "a290f39b-7f4c-4927-8fa3-3094bb02374c" (UID: "a290f39b-7f4c-4927-8fa3-3094bb02374c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.950845 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.950820    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-tmp-dir" pod "a290f39b-7f4c-4927-8fa3-3094bb02374c" (UID: "a290f39b-7f4c-4927-8fa3-3094bb02374c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.951040 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.950888    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-model-cache" pod "a290f39b-7f4c-4927-8fa3-3094bb02374c" (UID: "a290f39b-7f4c-4927-8fa3-3094bb02374c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.951274 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.951255    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.951347 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.951277    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgcs5\" (UniqueName: \"kubernetes.io/projected/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-kube-api-access-lgcs5\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.951347 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.951288    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.951347 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.951298    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/54698a98-165c-4f90-a571-6fee68c2d0f1-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.951347 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.951308    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.951565 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.951319    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e8cde83-0122-42ac-a2d5-e6c2e6d97703-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.951565 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.951378    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-456fh\" (UniqueName: \"kubernetes.io/projected/54698a98-165c-4f90-a571-6fee68c2d0f1-kube-api-access-456fh\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:08.952706 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.952682    4095 scope.go:122] "RemoveContainer" containerID="4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490"
Nov 01 12:57:08.953374 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:08.953261    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490\": container with ID starting with 4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490 not found: ID does not exist" containerID="4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490"
Nov 01 12:57:08.953374 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.953300    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490"} err="failed to get container status \"4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490\": rpc error: code = NotFound desc = could not find container \"4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490\": container with ID starting with 4e0bec04899161d747273d42999c2d2a4b051513ba46eb0dfd8207a32ec5c490 not found: ID does not exist"
Nov 01 12:57:08.954273 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.954238    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-dshm" pod "a290f39b-7f4c-4927-8fa3-3094bb02374c" (UID: "a290f39b-7f4c-4927-8fa3-3094bb02374c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:08.954769 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.954736    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a290f39b-7f4c-4927-8fa3-3094bb02374c-tls-certs" pod "a290f39b-7f4c-4927-8fa3-3094bb02374c" (UID: "a290f39b-7f4c-4927-8fa3-3094bb02374c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:57:08.955668 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:08.955642    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a290f39b-7f4c-4927-8fa3-3094bb02374c-kube-api-access-z8vfg" pod "a290f39b-7f4c-4927-8fa3-3094bb02374c" (UID: "a290f39b-7f4c-4927-8fa3-3094bb02374c"). InnerVolumeSpecName "kube-api-access-z8vfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:09.053004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.052958    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a290f39b-7f4c-4927-8fa3-3094bb02374c-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:09.053004 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.053004    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a290f39b-7f4c-4927-8fa3-3094bb02374c-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:09.053195 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.053018    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z8vfg\" (UniqueName: \"kubernetes.io/projected/a290f39b-7f4c-4927-8fa3-3094bb02374c-kube-api-access-z8vfg\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:09.195504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.195465    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"]
Nov 01 12:57:09.199492 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.199438    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4"]
Nov 01 12:57:09.357582 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.357548    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"]
Nov 01 12:57:09.364875 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.364833    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/distinct-w2-c7206f-kserve-58fd8bf9-lwczv"]
Nov 01 12:57:09.486267 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.486191    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"]
Nov 01 12:57:09.489357 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.489328    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-c7206f/simulator-w2-c7206f-kserve-849f5969b5-2n59s"]
Nov 01 12:57:09.621279 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:09.620402    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-c7206f/premium-w2-c7206f-kserve-75bd596d8-dkvv4" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:57:10.190378 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:10.190348    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" path="/var/lib/kubelet/pods/3e8cde83-0122-42ac-a2d5-e6c2e6d97703/volumes"
Nov 01 12:57:10.190830 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:10.190815    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" path="/var/lib/kubelet/pods/54698a98-165c-4f90-a571-6fee68c2d0f1/volumes"
Nov 01 12:57:10.191247 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:10.191234    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" path="/var/lib/kubelet/pods/a290f39b-7f4c-4927-8fa3-3094bb02374c/volumes"
Nov 01 12:57:12.032893 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:12.032858    4095 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6706ea89_410e_4c88_9a21_820078f1c1d6.slice/crio-conmon-8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 12:57:12.384012 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.381718    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:57:12.385254 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.385239    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:57:12.441141 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.441100    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:57:12.490131 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.490063    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:57:12.505346 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505321    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6706ea89-410e-4c88-9a21-820078f1c1d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6706ea89-410e-4c88-9a21-820078f1c1d6-istiod-ca-cert\") pod \"6706ea89-410e-4c88-9a21-820078f1c1d6\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") "
Nov 01 12:57:12.505473 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505391    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-certs\") pod \"6706ea89-410e-4c88-9a21-820078f1c1d6\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") "
Nov 01 12:57:12.505473 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505436    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-podinfo\") pod \"89c942a0-8f71-44e6-a432-a3f8a69ec808\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") "
Nov 01 12:57:12.505473 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505463    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-podinfo\") pod \"6706ea89-410e-4c88-9a21-820078f1c1d6\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") "
Nov 01 12:57:12.505643 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505509    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-socket\") pod \"6706ea89-410e-4c88-9a21-820078f1c1d6\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") "
Nov 01 12:57:12.505643 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505544    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-socket\") pod \"89c942a0-8f71-44e6-a432-a3f8a69ec808\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") "
Nov 01 12:57:12.505643 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505578    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-data\") pod \"6706ea89-410e-4c88-9a21-820078f1c1d6\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") "
Nov 01 12:57:12.505643 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505615    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-token\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-token\") pod \"6706ea89-410e-4c88-9a21-820078f1c1d6\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") "
Nov 01 12:57:12.505904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505707    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6706ea89-410e-4c88-9a21-820078f1c1d6-istiod-ca-cert" pod "6706ea89-410e-4c88-9a21-820078f1c1d6" (UID: "6706ea89-410e-4c88-9a21-820078f1c1d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:12.505904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505729    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-certs\") pod \"89c942a0-8f71-44e6-a432-a3f8a69ec808\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") "
Nov 01 12:57:12.505904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505792    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89c942a0-8f71-44e6-a432-a3f8a69ec808-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c942a0-8f71-44e6-a432-a3f8a69ec808-istiod-ca-cert\") pod \"89c942a0-8f71-44e6-a432-a3f8a69ec808\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") "
Nov 01 12:57:12.505904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505823    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-kube-api-access-grmks\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-kube-api-access-grmks\") pod \"6706ea89-410e-4c88-9a21-820078f1c1d6\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") "
Nov 01 12:57:12.505904 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505892    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-envoy\") pod \"89c942a0-8f71-44e6-a432-a3f8a69ec808\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") "
Nov 01 12:57:12.506177 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505897    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-socket" pod "89c942a0-8f71-44e6-a432-a3f8a69ec808" (UID: "89c942a0-8f71-44e6-a432-a3f8a69ec808"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.506177 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.505925    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-kube-api-access-z25rj\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-kube-api-access-z25rj\") pod \"89c942a0-8f71-44e6-a432-a3f8a69ec808\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") "
Nov 01 12:57:12.506323 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.506300    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-certs" pod "89c942a0-8f71-44e6-a432-a3f8a69ec808" (UID: "89c942a0-8f71-44e6-a432-a3f8a69ec808"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.506376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.506321    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89c942a0-8f71-44e6-a432-a3f8a69ec808-istiod-ca-cert" pod "89c942a0-8f71-44e6-a432-a3f8a69ec808" (UID: "89c942a0-8f71-44e6-a432-a3f8a69ec808"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:12.506376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.506331    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-data\") pod \"89c942a0-8f71-44e6-a432-a3f8a69ec808\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") "
Nov 01 12:57:12.506376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.506362    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-token\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-token\") pod \"89c942a0-8f71-44e6-a432-a3f8a69ec808\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") "
Nov 01 12:57:12.506519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.506428    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-envoy\") pod \"6706ea89-410e-4c88-9a21-820078f1c1d6\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") "
Nov 01 12:57:12.506519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.506460    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-credential-socket\") pod \"6706ea89-410e-4c88-9a21-820078f1c1d6\" (UID: \"6706ea89-410e-4c88-9a21-820078f1c1d6\") "
Nov 01 12:57:12.506519 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.506498    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-credential-socket\") pod \"89c942a0-8f71-44e6-a432-a3f8a69ec808\" (UID: \"89c942a0-8f71-44e6-a432-a3f8a69ec808\") "
Nov 01 12:57:12.506664 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.506573    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-data" pod "89c942a0-8f71-44e6-a432-a3f8a69ec808" (UID: "89c942a0-8f71-44e6-a432-a3f8a69ec808"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.506886 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.506862    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-credential-socket" pod "89c942a0-8f71-44e6-a432-a3f8a69ec808" (UID: "89c942a0-8f71-44e6-a432-a3f8a69ec808"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.507291 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.507264    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89c942a0-8f71-44e6-a432-a3f8a69ec808-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.507455 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.507295    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.507455 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.507310    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.507455 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.507325    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6706ea89-410e-4c88-9a21-820078f1c1d6-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.507455 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.507338    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.507455 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.507351    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.508398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.508257    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-data" pod "6706ea89-410e-4c88-9a21-820078f1c1d6" (UID: "6706ea89-410e-4c88-9a21-820078f1c1d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.508398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.508370    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-credential-socket" pod "6706ea89-410e-4c88-9a21-820078f1c1d6" (UID: "6706ea89-410e-4c88-9a21-820078f1c1d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.508398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.508377    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-socket" pod "6706ea89-410e-4c88-9a21-820078f1c1d6" (UID: "6706ea89-410e-4c88-9a21-820078f1c1d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.508909 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.508888    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-certs" pod "6706ea89-410e-4c88-9a21-820078f1c1d6" (UID: "6706ea89-410e-4c88-9a21-820078f1c1d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.509034 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.508935    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-kube-api-access-z25rj" pod "89c942a0-8f71-44e6-a432-a3f8a69ec808" (UID: "89c942a0-8f71-44e6-a432-a3f8a69ec808"). InnerVolumeSpecName "kube-api-access-z25rj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.510555 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.510531    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-envoy" pod "89c942a0-8f71-44e6-a432-a3f8a69ec808" (UID: "89c942a0-8f71-44e6-a432-a3f8a69ec808"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.510625 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.510571    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-token" pod "89c942a0-8f71-44e6-a432-a3f8a69ec808" (UID: "89c942a0-8f71-44e6-a432-a3f8a69ec808"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.511069 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.511034    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-kube-api-access-grmks" pod "6706ea89-410e-4c88-9a21-820078f1c1d6" (UID: "6706ea89-410e-4c88-9a21-820078f1c1d6"). InnerVolumeSpecName "kube-api-access-grmks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.511069 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.511046    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-podinfo" pod "89c942a0-8f71-44e6-a432-a3f8a69ec808" (UID: "89c942a0-8f71-44e6-a432-a3f8a69ec808"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:57:12.511190 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.511093    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-envoy" pod "6706ea89-410e-4c88-9a21-820078f1c1d6" (UID: "6706ea89-410e-4c88-9a21-820078f1c1d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.511190 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.511126    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-token" pod "6706ea89-410e-4c88-9a21-820078f1c1d6" (UID: "6706ea89-410e-4c88-9a21-820078f1c1d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.511331 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.511303    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-podinfo" pod "6706ea89-410e-4c88-9a21-820078f1c1d6" (UID: "6706ea89-410e-4c88-9a21-820078f1c1d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:57:12.608615 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608582    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-credential-socket\") pod \"57973d4e-e991-4011-ac52-eeb345a4e4de\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") "
Nov 01 12:57:12.608615 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608618    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-podinfo\") pod \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") "
Nov 01 12:57:12.608888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608653    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-certs\") pod \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") "
Nov 01 12:57:12.608888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608690    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-data\") pod \"57973d4e-e991-4011-ac52-eeb345a4e4de\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") "
Nov 01 12:57:12.608888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608775    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-kube-api-access-ljgzm\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-kube-api-access-ljgzm\") pod \"57973d4e-e991-4011-ac52-eeb345a4e4de\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") "
Nov 01 12:57:12.608888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608815    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istiod-ca-cert\") pod \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") "
Nov 01 12:57:12.608888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608846    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-socket\") pod \"57973d4e-e991-4011-ac52-eeb345a4e4de\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") "
Nov 01 12:57:12.608888 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608880    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-certs\") pod \"57973d4e-e991-4011-ac52-eeb345a4e4de\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") "
Nov 01 12:57:12.609304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608886    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-credential-socket" pod "57973d4e-e991-4011-ac52-eeb345a4e4de" (UID: "57973d4e-e991-4011-ac52-eeb345a4e4de"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.609304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608920    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-token\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-token\") pod \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") "
Nov 01 12:57:12.609304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.608984    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-podinfo\") pod \"57973d4e-e991-4011-ac52-eeb345a4e4de\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") "
Nov 01 12:57:12.609304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609025    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-kube-api-access-zhf6l\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-kube-api-access-zhf6l\") pod \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") "
Nov 01 12:57:12.609304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609024    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-data" pod "57973d4e-e991-4011-ac52-eeb345a4e4de" (UID: "57973d4e-e991-4011-ac52-eeb345a4e4de"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.609304 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609034    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-certs" pod "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" (UID: "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.610577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609371    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istiod-ca-cert" pod "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" (UID: "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:12.610577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609409    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-certs" pod "57973d4e-e991-4011-ac52-eeb345a4e4de" (UID: "57973d4e-e991-4011-ac52-eeb345a4e4de"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.610577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609422    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-socket" pod "57973d4e-e991-4011-ac52-eeb345a4e4de" (UID: "57973d4e-e991-4011-ac52-eeb345a4e4de"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.610577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609416    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-socket\") pod \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") "
Nov 01 12:57:12.610577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609611    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57973d4e-e991-4011-ac52-eeb345a4e4de-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973d4e-e991-4011-ac52-eeb345a4e4de-istiod-ca-cert\") pod \"57973d4e-e991-4011-ac52-eeb345a4e4de\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") "
Nov 01 12:57:12.610577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609687    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-socket" pod "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" (UID: "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.610577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.609919    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57973d4e-e991-4011-ac52-eeb345a4e4de-istiod-ca-cert" pod "57973d4e-e991-4011-ac52-eeb345a4e4de" (UID: "57973d4e-e991-4011-ac52-eeb345a4e4de"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:12.610577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.610094    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-credential-socket\") pod \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") "
Nov 01 12:57:12.610577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.610210    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-token\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-token\") pod \"57973d4e-e991-4011-ac52-eeb345a4e4de\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") "
Nov 01 12:57:12.611062 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.610586    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-data\") pod \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") "
Nov 01 12:57:12.611062 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.610647    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-credential-socket" pod "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" (UID: "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.611062 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.610683    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-data" pod "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" (UID: "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.611062 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.610730    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-envoy\") pod \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\" (UID: \"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a\") "
Nov 01 12:57:12.611062 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.610820    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-envoy\") pod \"57973d4e-e991-4011-ac52-eeb345a4e4de\" (UID: \"57973d4e-e991-4011-ac52-eeb345a4e4de\") "
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612280    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612339    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612360    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612413    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grmks\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-kube-api-access-grmks\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612428    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612507    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z25rj\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-kube-api-access-z25rj\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612527    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612542    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612557    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612583    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612601    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612615    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612630    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612645    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612659    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57973d4e-e991-4011-ac52-eeb345a4e4de-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612673    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89c942a0-8f71-44e6-a432-a3f8a69ec808-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612690    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612706    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612722    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612735    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612749    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.612792 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.612763    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6706ea89-410e-4c88-9a21-820078f1c1d6-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.613861 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.613423    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-podinfo" pod "57973d4e-e991-4011-ac52-eeb345a4e4de" (UID: "57973d4e-e991-4011-ac52-eeb345a4e4de"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:57:12.613861 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.613442    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-podinfo" pod "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" (UID: "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:57:12.614699 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.614518    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-kube-api-access-zhf6l" pod "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" (UID: "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a"). InnerVolumeSpecName "kube-api-access-zhf6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.614699 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.614668    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-token" pod "57973d4e-e991-4011-ac52-eeb345a4e4de" (UID: "57973d4e-e991-4011-ac52-eeb345a4e4de"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.615064 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.615007    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-kube-api-access-ljgzm" pod "57973d4e-e991-4011-ac52-eeb345a4e4de" (UID: "57973d4e-e991-4011-ac52-eeb345a4e4de"). InnerVolumeSpecName "kube-api-access-ljgzm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.615064 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.615054    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-envoy" pod "57973d4e-e991-4011-ac52-eeb345a4e4de" (UID: "57973d4e-e991-4011-ac52-eeb345a4e4de"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.615641 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.615135    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-envoy" pod "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" (UID: "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:12.615912 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.615878    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-token" pod "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" (UID: "4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:12.714291 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.714262    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ljgzm\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-kube-api-access-ljgzm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.714291 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.714288    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.714291 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.714298    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.714524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.714307    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zhf6l\" (UniqueName: \"kubernetes.io/projected/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-kube-api-access-zhf6l\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.714524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.714316    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.714524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.714324    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.714524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.714332    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57973d4e-e991-4011-ac52-eeb345a4e4de-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.714524 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.714342    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:12.824168 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.824088    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:57:12.824309 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.824158    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:57:12.910319 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.910261    4095 generic.go:356] "Generic (PLEG): container finished" podID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerID="162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c" exitCode=0
Nov 01 12:57:12.913035 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.910683    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" event={"ID":"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a","Type":"ContainerDied","Data":"162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c"}
Nov 01 12:57:12.913035 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.910752    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx" event={"ID":"4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a","Type":"ContainerDied","Data":"425ec346dac49c40a6a13886615f36b65a309465ae9bbbb9d94a4481fc8b1799"}
Nov 01 12:57:12.913035 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.910784    4095 scope.go:122] "RemoveContainer" containerID="162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c"
Nov 01 12:57:12.913035 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.910920    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"
Nov 01 12:57:12.913434 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.913327    4095 generic.go:356] "Generic (PLEG): container finished" podID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerID="850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970" exitCode=0
Nov 01 12:57:12.913487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.913434    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" event={"ID":"89c942a0-8f71-44e6-a432-a3f8a69ec808","Type":"ContainerDied","Data":"850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970"}
Nov 01 12:57:12.913487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.913480    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965" event={"ID":"89c942a0-8f71-44e6-a432-a3f8a69ec808","Type":"ContainerDied","Data":"43fc47d105b24f08b1b22d42d727c9a6aad8def0d812a6057465b1e002f1dae2"}
Nov 01 12:57:12.913592 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.913550    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"
Nov 01 12:57:12.923146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.923116    4095 generic.go:356] "Generic (PLEG): container finished" podID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerID="8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6" exitCode=0
Nov 01 12:57:12.923590 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.923560    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" event={"ID":"6706ea89-410e-4c88-9a21-820078f1c1d6","Type":"ContainerDied","Data":"8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6"}
Nov 01 12:57:12.923733 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.923618    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh" event={"ID":"6706ea89-410e-4c88-9a21-820078f1c1d6","Type":"ContainerDied","Data":"794b6949ad64770c2e742494f3424fc48e4d362fd7a63c0c12bbe7b3034cd194"}
Nov 01 12:57:12.923733 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.923697    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"
Nov 01 12:57:12.924310 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.924287    4095 scope.go:122] "RemoveContainer" containerID="162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c"
Nov 01 12:57:12.926733 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:12.926697    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c\": container with ID starting with 162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c not found: ID does not exist" containerID="162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c"
Nov 01 12:57:12.926814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.926744    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c"} err="failed to get container status \"162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c\": rpc error: code = NotFound desc = could not find container \"162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c\": container with ID starting with 162c731ae5379363bca8762f8fdd407ee51c501cb0040921addab4fc7d45610c not found: ID does not exist"
Nov 01 12:57:12.926814 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.926769    4095 scope.go:122] "RemoveContainer" containerID="850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970"
Nov 01 12:57:12.929079 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.929056    4095 generic.go:356] "Generic (PLEG): container finished" podID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerID="21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc" exitCode=0
Nov 01 12:57:12.929465 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.929113    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"
Nov 01 12:57:12.929655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.929130    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" event={"ID":"57973d4e-e991-4011-ac52-eeb345a4e4de","Type":"ContainerDied","Data":"21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc"}
Nov 01 12:57:12.929748 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.929695    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d" event={"ID":"57973d4e-e991-4011-ac52-eeb345a4e4de","Type":"ContainerDied","Data":"7f4486f5b18ded193631d7af2965d0980d382add4fa541d3a16eb5bb60ed1294"}
Nov 01 12:57:12.939282 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.939265    4095 scope.go:122] "RemoveContainer" containerID="850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970"
Nov 01 12:57:12.939647 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:12.939615    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970\": container with ID starting with 850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970 not found: ID does not exist" containerID="850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970"
Nov 01 12:57:12.939774 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.939657    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970"} err="failed to get container status \"850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970\": rpc error: code = NotFound desc = could not find container \"850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970\": container with ID starting with 850396f7e8d0958ca7bb9e7f57c4f9f9981048cb62b01b211cd3311c60a77970 not found: ID does not exist"
Nov 01 12:57:12.939774 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.939684    4095 scope.go:122] "RemoveContainer" containerID="8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6"
Nov 01 12:57:12.950181 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.950156    4095 scope.go:122] "RemoveContainer" containerID="8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6"
Nov 01 12:57:12.950458 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:12.950435    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6\": container with ID starting with 8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6 not found: ID does not exist" containerID="8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6"
Nov 01 12:57:12.950547 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.950467    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6"} err="failed to get container status \"8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6\": rpc error: code = NotFound desc = could not find container \"8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6\": container with ID starting with 8bf2a314e9ced52789fe7d5f112d059adef694fa51d8afe36fd6faa9b72154e6 not found: ID does not exist"
Nov 01 12:57:12.950547 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.950491    4095 scope.go:122] "RemoveContainer" containerID="21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc"
Nov 01 12:57:12.959829 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.959800    4095 scope.go:122] "RemoveContainer" containerID="21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc"
Nov 01 12:57:12.960126 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:12.960096    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc\": container with ID starting with 21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc not found: ID does not exist" containerID="21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc"
Nov 01 12:57:12.960201 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:12.960135    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc"} err="failed to get container status \"21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc\": rpc error: code = NotFound desc = could not find container \"21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc\": container with ID starting with 21d5758fe80fc40ce91a5cda201d65757f9a5a20b46113cf958398b0ac6875dc not found: ID does not exist"
Nov 01 12:57:13.077150 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:13.077063    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"]
Nov 01 12:57:13.080655 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:13.080629    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-ecd640-openshift-default-cbf985547-gjpxh"]
Nov 01 12:57:13.183594 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:13.183564    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"]
Nov 01 12:57:13.190025 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:13.189999    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-25jcx"]
Nov 01 12:57:13.295129 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:13.295092    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"]
Nov 01 12:57:13.300609 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:13.300580    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-pjg7d"]
Nov 01 12:57:13.404402 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:13.404369    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"]
Nov 01 12:57:13.412652 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:13.412621    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-2r965"]
Nov 01 12:57:14.198264 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.198231    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" path="/var/lib/kubelet/pods/4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a/volumes"
Nov 01 12:57:14.198767 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.198697    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" path="/var/lib/kubelet/pods/57973d4e-e991-4011-ac52-eeb345a4e4de/volumes"
Nov 01 12:57:14.199146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.199129    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" path="/var/lib/kubelet/pods/6706ea89-410e-4c88-9a21-820078f1c1d6/volumes"
Nov 01 12:57:14.199848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.199832    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" path="/var/lib/kubelet/pods/89c942a0-8f71-44e6-a432-a3f8a69ec808/volumes"
Nov 01 12:57:14.948031 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.948001    4095 generic.go:356] "Generic (PLEG): container finished" podID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerID="caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc" exitCode=0
Nov 01 12:57:14.948254 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.948079    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" event={"ID":"4dc78e78-3863-44a7-b0b1-5f96566e8ec8","Type":"ContainerDied","Data":"caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc"}
Nov 01 12:57:14.948601 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.948583    4095 scope.go:122] "RemoveContainer" containerID="caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc"
Nov 01 12:57:14.950322 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.950296    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:57:14.950388 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.950366    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:57:14.950427 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:14.950398    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:57:15.956044 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:15.955999    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" event={"ID":"4dc78e78-3863-44a7-b0b1-5f96566e8ec8","Type":"ContainerStarted","Data":"6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68"}
Nov 01 12:57:16.220056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:16.214417    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:57:16.220056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:16.214551    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 12:57:16.220056 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:16.214625    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 12:57:17.220052 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:17.214359    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 12:57:17.220052 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:17.214421    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 12:57:18.226998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:18.225345    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 12:57:18.226998 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:18.225406    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 12:57:19.137614 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:19.137582    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:57:19.223997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:19.212332    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 12:57:19.223997 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:19.212392    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 12:57:20.226996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:20.225342    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 12:57:20.226996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:20.225403    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 12:57:20.481764 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:20.481686    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:57:20.490380 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:20.490357    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 12:57:21.204482 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.204444    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"]
Nov 01 12:57:21.205601 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.205571    4095 cpu_manager.go:414] "containerMap: removing container" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy"
Nov 01 12:57:21.205754 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.205740    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy"
Nov 01 12:57:21.205877 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.205865    4095 cpu_manager.go:414] "containerMap: removing container" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy"
Nov 01 12:57:21.205960 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.205949    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy"
Nov 01 12:57:21.206093 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206081    4095 cpu_manager.go:414] "containerMap: removing container" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main"
Nov 01 12:57:21.206170 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206159    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main"
Nov 01 12:57:21.206260 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206250    4095 cpu_manager.go:414] "containerMap: removing container" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main"
Nov 01 12:57:21.206330 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206320    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main"
Nov 01 12:57:21.206418 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206408    4095 cpu_manager.go:414] "containerMap: removing container" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy"
Nov 01 12:57:21.206492 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206481    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy"
Nov 01 12:57:21.206572 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206563    4095 cpu_manager.go:414] "containerMap: removing container" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy"
Nov 01 12:57:21.206645 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206635    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy"
Nov 01 12:57:21.206750 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206738    4095 cpu_manager.go:414] "containerMap: removing container" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy"
Nov 01 12:57:21.206838 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206828    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy"
Nov 01 12:57:21.206927 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.206918    4095 cpu_manager.go:414] "containerMap: removing container" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main"
Nov 01 12:57:21.207038 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.207026    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main"
Nov 01 12:57:21.207119 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.207108    4095 cpu_manager.go:414] "containerMap: removing container" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy"
Nov 01 12:57:21.207202 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.207190    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy"
Nov 01 12:57:21.221996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.207409    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8cde83-0122-42ac-a2d5-e6c2e6d97703" containerName="main"
Nov 01 12:57:21.222273 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.222235    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="a290f39b-7f4c-4927-8fa3-3094bb02374c" containerName="main"
Nov 01 12:57:21.222445 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.222429    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy"
Nov 01 12:57:21.222550 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.222538    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="89c942a0-8f71-44e6-a432-a3f8a69ec808" containerName="istio-proxy"
Nov 01 12:57:21.222650 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.222639    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="6706ea89-410e-4c88-9a21-820078f1c1d6" containerName="istio-proxy"
Nov 01 12:57:21.222738 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.222718    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="54698a98-165c-4f90-a571-6fee68c2d0f1" containerName="main"
Nov 01 12:57:21.222827 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.222817    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="57973d4e-e991-4011-ac52-eeb345a4e4de" containerName="istio-proxy"
Nov 01 12:57:21.222920 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.222911    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8cbbd8-ddb5-4def-a36b-e4dac5aca08a" containerName="istio-proxy"
Nov 01 12:57:21.230989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.218371    4095 patch_prober.go:29] interesting pod/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 01 12:57:21.230989 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.227484    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 01 12:57:21.233667 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.233644    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.278995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.257906    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-certs\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.278995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.257953    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxkwg\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-kube-api-access-dxkwg\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.278995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.258117    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-data\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.278995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.258158    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-token\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.278995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.258240    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-envoy\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.278995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.258265    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-credential-socket\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.278995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.258328    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istiod-ca-cert\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.278995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.258412    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-socket\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.278995 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.258469    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-podinfo\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.290081 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.290053    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-72078a-openshift-default-dockercfg-hd4m2\""
Nov 01 12:57:21.337353 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.337320    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"]
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.360342    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-data\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.360413    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-token\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.360476    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-envoy\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.360543    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-credential-socket\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.360614    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istiod-ca-cert\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.360679    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-socket\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.360737    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-podinfo\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.360861    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-certs\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.360891    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxkwg\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-kube-api-access-dxkwg\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362017 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.361590    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-credential-socket\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362651 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.362410    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istiod-ca-cert\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.362706 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.362658    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-socket\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.363328 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.363308    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-certs\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.363476 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.363359    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-data\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.366218 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.366177    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-envoy\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.366446 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.366424    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-podinfo\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.424137 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.424109    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-token\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.424569 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.424544    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxkwg\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-kube-api-access-dxkwg\") pod \"e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") " pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:21.592433 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:21.592353    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:22.048636 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.048608    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"]
Nov 01 12:57:22.048775 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:57:22.048755    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcec0037a_c467_4cb4_bb5b_d556ff5e7cf6.slice/crio-16d7bca712f0c405aa43fc7dff94e35737f6b1c96dfa0e277705abfee8c89de9 WatchSource:0}: Error finding container 16d7bca712f0c405aa43fc7dff94e35737f6b1c96dfa0e277705abfee8c89de9: Status 404 returned error can't find the container with id 16d7bca712f0c405aa43fc7dff94e35737f6b1c96dfa0e277705abfee8c89de9
Nov 01 12:57:22.051460 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.051430    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:57:22.051555 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.051494    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:57:22.051555 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.051520    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:57:22.199784 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.199744    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"]
Nov 01 12:57:22.202500 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.201122    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy" containerID="cri-o://5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7" gracePeriod=30
Nov 01 12:57:22.214426 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.214402    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:57:22.215755 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.215728    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:57:22.216602 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.216572    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 12:57:22.522403 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.522372    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"]
Nov 01 12:57:22.522833 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:22.522573    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy" containerID="cri-o://0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf" gracePeriod=30
Nov 01 12:57:23.004596 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:23.004550    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" event={"ID":"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6","Type":"ContainerStarted","Data":"d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52"}
Nov 01 12:57:23.004871 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:23.004622    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" event={"ID":"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6","Type":"ContainerStarted","Data":"16d7bca712f0c405aa43fc7dff94e35737f6b1c96dfa0e277705abfee8c89de9"}
Nov 01 12:57:23.129766 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:23.129714    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" podStartSLOduration=2.12969521 podStartE2EDuration="2.12969521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:57:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:57:23.127553037 +0000 UTC m=+2103.045348702" watchObservedRunningTime="2026-11-01 12:57:23.12969521 +0000 UTC m=+2103.047490876"
Nov 01 12:57:23.593141 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:23.593107    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:23.598432 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:23.598410    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:23.622550 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:23.622527    4095 patch_prober.go:29] interesting pod/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 12:57:23.622677 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:23.622572    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 12:57:24.010885 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:24.010846    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:24.011822 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:24.011799    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:57:27.464984 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.464946    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:57:27.647491 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.647465    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-certs\") pod \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") "
Nov 01 12:57:27.647609 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.647582    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-data\") pod \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") "
Nov 01 12:57:27.647682 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.647626    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-socket\") pod \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") "
Nov 01 12:57:27.647845 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.647719    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-credential-socket\") pod \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") "
Nov 01 12:57:27.647845 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.647760    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-certs" pod "0ed0cde7-7d50-43da-b493-bad0c76f03e7" (UID: "0ed0cde7-7d50-43da-b493-bad0c76f03e7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.647845 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.647771    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-podinfo\") pod \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") "
Nov 01 12:57:27.648045 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.647850    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-data" pod "0ed0cde7-7d50-43da-b493-bad0c76f03e7" (UID: "0ed0cde7-7d50-43da-b493-bad0c76f03e7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.648045 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.647914    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-token\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-token\") pod \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") "
Nov 01 12:57:27.648153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.648021    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-socket" pod "0ed0cde7-7d50-43da-b493-bad0c76f03e7" (UID: "0ed0cde7-7d50-43da-b493-bad0c76f03e7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.648153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.648086    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-envoy\") pod \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") "
Nov 01 12:57:27.648153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.648099    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-credential-socket" pod "0ed0cde7-7d50-43da-b493-bad0c76f03e7" (UID: "0ed0cde7-7d50-43da-b493-bad0c76f03e7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.648153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.648140    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istiod-ca-cert\") pod \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") "
Nov 01 12:57:27.648342 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.648209    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-kube-api-access-5n2fh\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-kube-api-access-5n2fh\") pod \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\" (UID: \"0ed0cde7-7d50-43da-b493-bad0c76f03e7\") "
Nov 01 12:57:27.648846 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.648822    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istiod-ca-cert" pod "0ed0cde7-7d50-43da-b493-bad0c76f03e7" (UID: "0ed0cde7-7d50-43da-b493-bad0c76f03e7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:27.649002 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.648855    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.649138 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.649115    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.649257 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.649142    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.649257 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.649159    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.650476 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.650447    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-envoy" pod "0ed0cde7-7d50-43da-b493-bad0c76f03e7" (UID: "0ed0cde7-7d50-43da-b493-bad0c76f03e7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.650708 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.650682    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-kube-api-access-5n2fh" pod "0ed0cde7-7d50-43da-b493-bad0c76f03e7" (UID: "0ed0cde7-7d50-43da-b493-bad0c76f03e7"). InnerVolumeSpecName "kube-api-access-5n2fh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:27.650927 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.650911    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-token" pod "0ed0cde7-7d50-43da-b493-bad0c76f03e7" (UID: "0ed0cde7-7d50-43da-b493-bad0c76f03e7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:27.651427 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.651407    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-podinfo" pod "0ed0cde7-7d50-43da-b493-bad0c76f03e7" (UID: "0ed0cde7-7d50-43da-b493-bad0c76f03e7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:57:27.752172 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.751544    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.752172 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.751586    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.752172 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.751600    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.752172 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.751615    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5n2fh\" (UniqueName: \"kubernetes.io/projected/0ed0cde7-7d50-43da-b493-bad0c76f03e7-kube-api-access-5n2fh\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.752172 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.751630    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ed0cde7-7d50-43da-b493-bad0c76f03e7-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.806487 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.806460    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:57:27.954058 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954025    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-podinfo\") pod \"453656f7-ef59-46a5-925f-37c8190d6b0f\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") "
Nov 01 12:57:27.954266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954078    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-credential-socket\") pod \"453656f7-ef59-46a5-925f-37c8190d6b0f\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") "
Nov 01 12:57:27.954266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954100    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-envoy\") pod \"453656f7-ef59-46a5-925f-37c8190d6b0f\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") "
Nov 01 12:57:27.954266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954124    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-data\") pod \"453656f7-ef59-46a5-925f-37c8190d6b0f\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") "
Nov 01 12:57:27.954266 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954264    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-socket\") pod \"453656f7-ef59-46a5-925f-37c8190d6b0f\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") "
Nov 01 12:57:27.954494 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954352    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/453656f7-ef59-46a5-925f-37c8190d6b0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453656f7-ef59-46a5-925f-37c8190d6b0f-istiod-ca-cert\") pod \"453656f7-ef59-46a5-925f-37c8190d6b0f\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") "
Nov 01 12:57:27.954494 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954400    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-token\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-token\") pod \"453656f7-ef59-46a5-925f-37c8190d6b0f\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") "
Nov 01 12:57:27.954494 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954438    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-certs\") pod \"453656f7-ef59-46a5-925f-37c8190d6b0f\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") "
Nov 01 12:57:27.954494 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954471    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-kube-api-access-bhfvc\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-kube-api-access-bhfvc\") pod \"453656f7-ef59-46a5-925f-37c8190d6b0f\" (UID: \"453656f7-ef59-46a5-925f-37c8190d6b0f\") "
Nov 01 12:57:27.954699 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.954494    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-data" pod "453656f7-ef59-46a5-925f-37c8190d6b0f" (UID: "453656f7-ef59-46a5-925f-37c8190d6b0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.955125 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.955095    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-socket" pod "453656f7-ef59-46a5-925f-37c8190d6b0f" (UID: "453656f7-ef59-46a5-925f-37c8190d6b0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.955352 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.955325    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-certs" pod "453656f7-ef59-46a5-925f-37c8190d6b0f" (UID: "453656f7-ef59-46a5-925f-37c8190d6b0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.955433 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.955409    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-credential-socket" pod "453656f7-ef59-46a5-925f-37c8190d6b0f" (UID: "453656f7-ef59-46a5-925f-37c8190d6b0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.955433 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.955418    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/453656f7-ef59-46a5-925f-37c8190d6b0f-istiod-ca-cert" pod "453656f7-ef59-46a5-925f-37c8190d6b0f" (UID: "453656f7-ef59-46a5-925f-37c8190d6b0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:57:27.955563 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.955538    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.955563 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.955563    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:27.956947 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.956921    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-podinfo" pod "453656f7-ef59-46a5-925f-37c8190d6b0f" (UID: "453656f7-ef59-46a5-925f-37c8190d6b0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:57:27.957060 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.956928    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-envoy" pod "453656f7-ef59-46a5-925f-37c8190d6b0f" (UID: "453656f7-ef59-46a5-925f-37c8190d6b0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:57:27.957060 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.956984    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-kube-api-access-bhfvc" pod "453656f7-ef59-46a5-925f-37c8190d6b0f" (UID: "453656f7-ef59-46a5-925f-37c8190d6b0f"). InnerVolumeSpecName "kube-api-access-bhfvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:27.957222 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:27.957200    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-token" pod "453656f7-ef59-46a5-925f-37c8190d6b0f" (UID: "453656f7-ef59-46a5-925f-37c8190d6b0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:57:28.034327 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.034292    4095 generic.go:356] "Generic (PLEG): container finished" podID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerID="5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7" exitCode=0
Nov 01 12:57:28.034536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.034350    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"
Nov 01 12:57:28.034536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.034379    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" event={"ID":"0ed0cde7-7d50-43da-b493-bad0c76f03e7","Type":"ContainerDied","Data":"5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7"}
Nov 01 12:57:28.034536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.034444    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk" event={"ID":"0ed0cde7-7d50-43da-b493-bad0c76f03e7","Type":"ContainerDied","Data":"fa776965b87b01c40af54c1b98c4ff21d53f3a0a3d70fb26da218515de1206b3"}
Nov 01 12:57:28.034536 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.034460    4095 scope.go:122] "RemoveContainer" containerID="5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7"
Nov 01 12:57:28.035838 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.035815    4095 generic.go:356] "Generic (PLEG): container finished" podID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerID="0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf" exitCode=0
Nov 01 12:57:28.035994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.035846    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" event={"ID":"453656f7-ef59-46a5-925f-37c8190d6b0f","Type":"ContainerDied","Data":"0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf"}
Nov 01 12:57:28.035994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.035868    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"
Nov 01 12:57:28.035994 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.035894    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x" event={"ID":"453656f7-ef59-46a5-925f-37c8190d6b0f","Type":"ContainerDied","Data":"747954627de71c46a9200b57dac54070ec1c0c88bfa613d55ba6326f1e5df560"}
Nov 01 12:57:28.049505 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.049262    4095 scope.go:122] "RemoveContainer" containerID="5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7"
Nov 01 12:57:28.049998 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:28.049960    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7\": container with ID starting with 5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7 not found: ID does not exist" containerID="5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7"
Nov 01 12:57:28.050094 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.050013    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7"} err="failed to get container status \"5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7\": rpc error: code = NotFound desc = could not find container \"5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7\": container with ID starting with 5ebd81296149d4cec880db413410da2c695e66ac86aa08fb19c6ff47b2825ea7 not found: ID does not exist"
Nov 01 12:57:28.050094 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.050039    4095 scope.go:122] "RemoveContainer" containerID="0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf"
Nov 01 12:57:28.057123 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.057098    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:28.057123 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.057120    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:28.057247 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.057130    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:28.057247 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.057139    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/453656f7-ef59-46a5-925f-37c8190d6b0f-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:28.057247 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.057147    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:28.057247 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.057157    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/453656f7-ef59-46a5-925f-37c8190d6b0f-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:28.057247 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.057165    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhfvc\" (UniqueName: \"kubernetes.io/projected/453656f7-ef59-46a5-925f-37c8190d6b0f-kube-api-access-bhfvc\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:57:28.059574 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.059555    4095 scope.go:122] "RemoveContainer" containerID="0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf"
Nov 01 12:57:28.059861 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:28.059843    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf\": container with ID starting with 0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf not found: ID does not exist" containerID="0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf"
Nov 01 12:57:28.059922 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.059869    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf"} err="failed to get container status \"0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf\": rpc error: code = NotFound desc = could not find container \"0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf\": container with ID starting with 0c8a8c43832e1ec74d3273d2ef48a4900f77556a691b0cde57c7e3b1b0a1facf not found: ID does not exist"
Nov 01 12:57:28.200434 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.200404    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"]
Nov 01 12:57:28.203398 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.203373    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-49sgk"]
Nov 01 12:57:28.310371 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.310293    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"]
Nov 01 12:57:28.322113 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:28.322087    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-9514b0-openshift-default-957cbc7d9-4pr2x"]
Nov 01 12:57:30.193934 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.193899    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" path="/var/lib/kubelet/pods/0ed0cde7-7d50-43da-b493-bad0c76f03e7/volumes"
Nov 01 12:57:30.194529 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.194511    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" path="/var/lib/kubelet/pods/453656f7-ef59-46a5-925f-37c8190d6b0f/volumes"
Nov 01 12:57:30.194881 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.194863    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"]
Nov 01 12:57:30.195529 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.195509    4095 cpu_manager.go:414] "containerMap: removing container" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy"
Nov 01 12:57:30.195529 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.195528    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy"
Nov 01 12:57:30.195764 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.195586    4095 cpu_manager.go:414] "containerMap: removing container" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy"
Nov 01 12:57:30.195764 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.195595    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy"
Nov 01 12:57:30.195916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.195771    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="1064c601-a2a0-42af-9fcc-5886f08617e4" containerName="istio-proxy"
Nov 01 12:57:30.195916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.195791    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ed0cde7-7d50-43da-b493-bad0c76f03e7" containerName="istio-proxy"
Nov 01 12:57:30.195916 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.195808    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="453656f7-ef59-46a5-925f-37c8190d6b0f" containerName="istio-proxy"
Nov 01 12:57:30.201529 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.201504    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:30.226946 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.226916    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"]
Nov 01 12:57:30.271124 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.271094    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-72078a\""
Nov 01 12:57:30.271124 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.271111    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-72078a-dockercfg-txktk\""
Nov 01 12:57:30.297784 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.297749    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004f8295-ef7a-4c55-b1fd-317c99992416-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm\" (UID: \"004f8295-ef7a-4c55-b1fd-317c99992416\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:30.298015 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.297996    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmxtw\" (UniqueName: \"kubernetes.io/projected/004f8295-ef7a-4c55-b1fd-317c99992416-kube-api-access-hmxtw\") pod \"payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm\" (UID: \"004f8295-ef7a-4c55-b1fd-317c99992416\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:30.399155 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.399121    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmxtw\" (UniqueName: \"kubernetes.io/projected/004f8295-ef7a-4c55-b1fd-317c99992416-kube-api-access-hmxtw\") pod \"payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm\" (UID: \"004f8295-ef7a-4c55-b1fd-317c99992416\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:30.399362 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.399236    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004f8295-ef7a-4c55-b1fd-317c99992416-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm\" (UID: \"004f8295-ef7a-4c55-b1fd-317c99992416\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:30.400000 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.399937    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004f8295-ef7a-4c55-b1fd-317c99992416-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm\" (UID: \"004f8295-ef7a-4c55-b1fd-317c99992416\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:30.454286 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.454218    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmxtw\" (UniqueName: \"kubernetes.io/projected/004f8295-ef7a-4c55-b1fd-317c99992416-kube-api-access-hmxtw\") pod \"payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm\" (UID: \"004f8295-ef7a-4c55-b1fd-317c99992416\") " pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:30.518177 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.518145    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:30.901485 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:30.901452    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"]
Nov 01 12:57:30.901616 ip-10-0-140-168 kubenswrapper[4095]: W1101 12:57:30.901485    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004f8295_ef7a_4c55_b1fd_317c99992416.slice/crio-a8433989a72f3238732d9e4f3eaa37173be2afb30600fe6dfe1880fb95d395d1 WatchSource:0}: Error finding container a8433989a72f3238732d9e4f3eaa37173be2afb30600fe6dfe1880fb95d395d1: Status 404 returned error can't find the container with id a8433989a72f3238732d9e4f3eaa37173be2afb30600fe6dfe1880fb95d395d1
Nov 01 12:57:31.058902 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:31.058859    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm" event={"ID":"004f8295-ef7a-4c55-b1fd-317c99992416","Type":"ContainerStarted","Data":"c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69"}
Nov 01 12:57:31.059107 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:31.058959    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm" event={"ID":"004f8295-ef7a-4c55-b1fd-317c99992416","Type":"ContainerStarted","Data":"a8433989a72f3238732d9e4f3eaa37173be2afb30600fe6dfe1880fb95d395d1"}
Nov 01 12:57:31.059186 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:31.059139    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:31.172949 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:31.172841    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm" podStartSLOduration=1.1728283369999999 podStartE2EDuration="1.172828337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 12:57:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 12:57:31.171061055 +0000 UTC m=+2111.088856720" watchObservedRunningTime="2026-11-01 12:57:31.172828337 +0000 UTC m=+2111.090624003"
Nov 01 12:57:42.068154 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:57:42.068124    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 12:57:49.115594 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:57:49.115555    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:58:19.114223 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:58:19.114188    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:58:45.395685 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:45.395654    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"]
Nov 01 12:58:49.114341 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:58:49.114299    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:58:50.396747 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.396689    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4" podUID="a9496594-0329-4d04-afc7-7fa862ff4d3b" containerName="payload-processing" containerID="cri-o://276373fa6b60cd1f49bb55e79bf38b63f119915050cbbc005416dc0c7943f321" gracePeriod=25
Nov 01 12:58:50.567424 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.567393    4095 generic.go:356] "Generic (PLEG): container finished" podID="a9496594-0329-4d04-afc7-7fa862ff4d3b" containerID="276373fa6b60cd1f49bb55e79bf38b63f119915050cbbc005416dc0c7943f321" exitCode=0
Nov 01 12:58:50.567577 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.567466    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4" event={"ID":"a9496594-0329-4d04-afc7-7fa862ff4d3b","Type":"ContainerDied","Data":"276373fa6b60cd1f49bb55e79bf38b63f119915050cbbc005416dc0c7943f321"}
Nov 01 12:58:50.675466 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.675443    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:58:50.697454 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.697428    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9496594-0329-4d04-afc7-7fa862ff4d3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9496594-0329-4d04-afc7-7fa862ff4d3b-plugins-config-volume\") pod \"a9496594-0329-4d04-afc7-7fa862ff4d3b\" (UID: \"a9496594-0329-4d04-afc7-7fa862ff4d3b\") "
Nov 01 12:58:50.697579 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.697517    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9496594-0329-4d04-afc7-7fa862ff4d3b-kube-api-access-jlv59\" (UniqueName: \"kubernetes.io/projected/a9496594-0329-4d04-afc7-7fa862ff4d3b-kube-api-access-jlv59\") pod \"a9496594-0329-4d04-afc7-7fa862ff4d3b\" (UID: \"a9496594-0329-4d04-afc7-7fa862ff4d3b\") "
Nov 01 12:58:50.697816 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.697792    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9496594-0329-4d04-afc7-7fa862ff4d3b-plugins-config-volume" pod "a9496594-0329-4d04-afc7-7fa862ff4d3b" (UID: "a9496594-0329-4d04-afc7-7fa862ff4d3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:58:50.699535 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.699511    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9496594-0329-4d04-afc7-7fa862ff4d3b-kube-api-access-jlv59" pod "a9496594-0329-4d04-afc7-7fa862ff4d3b" (UID: "a9496594-0329-4d04-afc7-7fa862ff4d3b"). InnerVolumeSpecName "kube-api-access-jlv59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:58:50.799561 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.799529    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlv59\" (UniqueName: \"kubernetes.io/projected/a9496594-0329-4d04-afc7-7fa862ff4d3b-kube-api-access-jlv59\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:50.799561 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:50.799563    4095 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9496594-0329-4d04-afc7-7fa862ff4d3b-plugins-config-volume\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:58:51.575040 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:51.575014    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"
Nov 01 12:58:51.575505 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:51.575004    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4" event={"ID":"a9496594-0329-4d04-afc7-7fa862ff4d3b","Type":"ContainerDied","Data":"8707993e98b795e8229a89bb763a28abaf48a7c78cc30aa2a5e923a1b15cb9f2"}
Nov 01 12:58:51.575505 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:51.575157    4095 scope.go:122] "RemoveContainer" containerID="276373fa6b60cd1f49bb55e79bf38b63f119915050cbbc005416dc0c7943f321"
Nov 01 12:58:51.733504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:51.733470    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"]
Nov 01 12:58:51.780743 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:51.780712    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-0cd09f-cc47fcb9-688n4"]
Nov 01 12:58:52.190348 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:58:52.190313    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9496594-0329-4d04-afc7-7fa862ff4d3b" path="/var/lib/kubelet/pods/a9496594-0329-4d04-afc7-7fa862ff4d3b/volumes"
Nov 01 12:59:02.540589 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.540509    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"]
Nov 01 12:59:02.541104 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.540730    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs" podUID="9d05dbc0-db8f-4084-874e-93b23b734511" containerName="maas-api" containerID="cri-o://9c1ae56ff75aa531d58dc64486ae3ebcb893a05f680fb6669eddcd7c1e153a96" gracePeriod=30
Nov 01 12:59:02.635260 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.635225    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs" podUID="9d05dbc0-db8f-4084-874e-93b23b734511" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.44:8443/health\": dial tcp 10.134.0.44:8443: connect: connection refused"
Nov 01 12:59:02.663678 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.663653    4095 generic.go:356] "Generic (PLEG): container finished" podID="9d05dbc0-db8f-4084-874e-93b23b734511" containerID="9c1ae56ff75aa531d58dc64486ae3ebcb893a05f680fb6669eddcd7c1e153a96" exitCode=0
Nov 01 12:59:02.663789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.663710    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs" event={"ID":"9d05dbc0-db8f-4084-874e-93b23b734511","Type":"ContainerDied","Data":"9c1ae56ff75aa531d58dc64486ae3ebcb893a05f680fb6669eddcd7c1e153a96"}
Nov 01 12:59:02.823008 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.822965    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:59:02.991166 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.991130    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d05dbc0-db8f-4084-874e-93b23b734511-kube-api-access-kj9bf\" (UniqueName: \"kubernetes.io/projected/9d05dbc0-db8f-4084-874e-93b23b734511-kube-api-access-kj9bf\") pod \"9d05dbc0-db8f-4084-874e-93b23b734511\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") "
Nov 01 12:59:02.991360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.991213    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-metrics-certs\") pod \"9d05dbc0-db8f-4084-874e-93b23b734511\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") "
Nov 01 12:59:02.991360 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.991334    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls\") pod \"9d05dbc0-db8f-4084-874e-93b23b734511\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") "
Nov 01 12:59:02.991474 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.991449    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d05dbc0-db8f-4084-874e-93b23b734511-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d05dbc0-db8f-4084-874e-93b23b734511-openshift-service-ca\") pod \"9d05dbc0-db8f-4084-874e-93b23b734511\" (UID: \"9d05dbc0-db8f-4084-874e-93b23b734511\") "
Nov 01 12:59:02.991866 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.991827    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d05dbc0-db8f-4084-874e-93b23b734511-openshift-service-ca" pod "9d05dbc0-db8f-4084-874e-93b23b734511" (UID: "9d05dbc0-db8f-4084-874e-93b23b734511"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:59:02.992053 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.992034    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d05dbc0-db8f-4084-874e-93b23b734511-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:02.993409 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.993384    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-metrics-certs" pod "9d05dbc0-db8f-4084-874e-93b23b734511" (UID: "9d05dbc0-db8f-4084-874e-93b23b734511"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:59:02.993497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.993421    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls" pod "9d05dbc0-db8f-4084-874e-93b23b734511" (UID: "9d05dbc0-db8f-4084-874e-93b23b734511"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:59:02.993497 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:02.993432    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d05dbc0-db8f-4084-874e-93b23b734511-kube-api-access-kj9bf" pod "9d05dbc0-db8f-4084-874e-93b23b734511" (UID: "9d05dbc0-db8f-4084-874e-93b23b734511"). InnerVolumeSpecName "kube-api-access-kj9bf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:03.100935 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:03.095201    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kj9bf\" (UniqueName: \"kubernetes.io/projected/9d05dbc0-db8f-4084-874e-93b23b734511-kube-api-access-kj9bf\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:03.100935 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:03.095244    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:03.100935 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:03.095260    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d05dbc0-db8f-4084-874e-93b23b734511-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:03.673132 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:03.673090    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs" event={"ID":"9d05dbc0-db8f-4084-874e-93b23b734511","Type":"ContainerDied","Data":"53c391fb0613fd698d1313dcf7f70a16ee15b653bc40389f33870119d20afd66"}
Nov 01 12:59:03.673717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:03.673183    4095 scope.go:122] "RemoveContainer" containerID="9c1ae56ff75aa531d58dc64486ae3ebcb893a05f680fb6669eddcd7c1e153a96"
Nov 01 12:59:03.673717 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:03.673108    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"
Nov 01 12:59:03.830343 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:03.830308    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"]
Nov 01 12:59:03.879530 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:03.879496    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-c7206f-656cb4767f-t6mxs"]
Nov 01 12:59:04.190240 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:04.190206    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d05dbc0-db8f-4084-874e-93b23b734511" path="/var/lib/kubelet/pods/9d05dbc0-db8f-4084-874e-93b23b734511/volumes"
Nov 01 12:59:12.109789 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:12.109753    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"]
Nov 01 12:59:12.110245 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:12.110023    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy" containerID="cri-o://098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0" gracePeriod=30
Nov 01 12:59:15.468897 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:15.468863    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"]
Nov 01 12:59:15.516762 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:15.516731    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"]
Nov 01 12:59:17.402964 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.402941    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:59:17.483205 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483127    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-envoy\") pod \"a17bdbf6-1545-43fd-8053-92a61f75edc1\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") "
Nov 01 12:59:17.483205 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483172    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-credential-socket\") pod \"a17bdbf6-1545-43fd-8053-92a61f75edc1\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") "
Nov 01 12:59:17.483504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483241    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-socket\") pod \"a17bdbf6-1545-43fd-8053-92a61f75edc1\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") "
Nov 01 12:59:17.483504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483267    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-token\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-token\") pod \"a17bdbf6-1545-43fd-8053-92a61f75edc1\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") "
Nov 01 12:59:17.483504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483408    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-data\") pod \"a17bdbf6-1545-43fd-8053-92a61f75edc1\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") "
Nov 01 12:59:17.483504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483449    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-kube-api-access-btr6j\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-kube-api-access-btr6j\") pod \"a17bdbf6-1545-43fd-8053-92a61f75edc1\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") "
Nov 01 12:59:17.483504 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483502    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-podinfo\") pod \"a17bdbf6-1545-43fd-8053-92a61f75edc1\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") "
Nov 01 12:59:17.483768 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483558    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a17bdbf6-1545-43fd-8053-92a61f75edc1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17bdbf6-1545-43fd-8053-92a61f75edc1-istiod-ca-cert\") pod \"a17bdbf6-1545-43fd-8053-92a61f75edc1\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") "
Nov 01 12:59:17.483768 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483556    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-socket" pod "a17bdbf6-1545-43fd-8053-92a61f75edc1" (UID: "a17bdbf6-1545-43fd-8053-92a61f75edc1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.483768 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483570    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-credential-socket" pod "a17bdbf6-1545-43fd-8053-92a61f75edc1" (UID: "a17bdbf6-1545-43fd-8053-92a61f75edc1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.483768 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483608    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-certs\") pod \"a17bdbf6-1545-43fd-8053-92a61f75edc1\" (UID: \"a17bdbf6-1545-43fd-8053-92a61f75edc1\") "
Nov 01 12:59:17.483996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483879    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-certs" pod "a17bdbf6-1545-43fd-8053-92a61f75edc1" (UID: "a17bdbf6-1545-43fd-8053-92a61f75edc1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.483996 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.483944    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a17bdbf6-1545-43fd-8053-92a61f75edc1-istiod-ca-cert" pod "a17bdbf6-1545-43fd-8053-92a61f75edc1" (UID: "a17bdbf6-1545-43fd-8053-92a61f75edc1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:59:17.484103 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.484015    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-data" pod "a17bdbf6-1545-43fd-8053-92a61f75edc1" (UID: "a17bdbf6-1545-43fd-8053-92a61f75edc1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.484209 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.484186    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.484276 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.484220    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a17bdbf6-1545-43fd-8053-92a61f75edc1-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.484276 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.484237    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.484276 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.484250    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.484276 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.484264    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.485660 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.485631    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-envoy" pod "a17bdbf6-1545-43fd-8053-92a61f75edc1" (UID: "a17bdbf6-1545-43fd-8053-92a61f75edc1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:17.485782 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.485666    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-kube-api-access-btr6j" pod "a17bdbf6-1545-43fd-8053-92a61f75edc1" (UID: "a17bdbf6-1545-43fd-8053-92a61f75edc1"). InnerVolumeSpecName "kube-api-access-btr6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:17.485849 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.485819    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-token" pod "a17bdbf6-1545-43fd-8053-92a61f75edc1" (UID: "a17bdbf6-1545-43fd-8053-92a61f75edc1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:17.486103 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.486086    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-podinfo" pod "a17bdbf6-1545-43fd-8053-92a61f75edc1" (UID: "a17bdbf6-1545-43fd-8053-92a61f75edc1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 12:59:17.586146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.586113    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.586146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.586139    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.586146 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.586150    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btr6j\" (UniqueName: \"kubernetes.io/projected/a17bdbf6-1545-43fd-8053-92a61f75edc1-kube-api-access-btr6j\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.586459 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.586161    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a17bdbf6-1545-43fd-8053-92a61f75edc1-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:17.759662 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.759561    4095 generic.go:356] "Generic (PLEG): container finished" podID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerID="098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0" exitCode=0
Nov 01 12:59:17.759662 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.759609    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"
Nov 01 12:59:17.759873 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.759652    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" event={"ID":"a17bdbf6-1545-43fd-8053-92a61f75edc1","Type":"ContainerDied","Data":"098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0"}
Nov 01 12:59:17.759873 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.759710    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg" event={"ID":"a17bdbf6-1545-43fd-8053-92a61f75edc1","Type":"ContainerDied","Data":"6f93621a4343554e090603cdc746a33a1c5bac02b4c3719ebffd93b5be2e8e93"}
Nov 01 12:59:17.759873 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.759731    4095 scope.go:122] "RemoveContainer" containerID="098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0"
Nov 01 12:59:17.770702 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.770680    4095 scope.go:122] "RemoveContainer" containerID="098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0"
Nov 01 12:59:17.770998 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:59:17.770957    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0\": container with ID starting with 098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0 not found: ID does not exist" containerID="098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0"
Nov 01 12:59:17.771091 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.771000    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0"} err="failed to get container status \"098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0\": rpc error: code = NotFound desc = could not find container \"098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0\": container with ID starting with 098d09492310533b14382d1fdbec5112a3961023d3ad579ff4b9343c376ccef0 not found: ID does not exist"
Nov 01 12:59:17.927533 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.927497    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"]
Nov 01 12:59:17.936114 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:17.936080    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-c7206f-openshift-default-856d699b69-ck9gg"]
Nov 01 12:59:18.192335 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:18.192294    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" path="/var/lib/kubelet/pods/a17bdbf6-1545-43fd-8053-92a61f75edc1/volumes"
Nov 01 12:59:19.135533 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:59:19.135490    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:59:25.051131 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:25.051087    4095 patch_prober.go:29] interesting pod/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 12:59:25.051563 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:25.051165    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 12:59:26.777211 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:26.777177    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"]
Nov 01 12:59:27.822262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:27.822161    4095 generic.go:356] "Generic (PLEG): container finished" podID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerID="d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52" exitCode=0
Nov 01 12:59:27.822262 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:27.822218    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" event={"ID":"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6","Type":"ContainerDied","Data":"d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52"}
Nov 01 12:59:27.822856 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:27.822729    4095 scope.go:122] "RemoveContainer" containerID="d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52"
Nov 01 12:59:27.825941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:27.825162    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:59:27.825941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:27.825371    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:59:27.825941 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:27.825473    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 12:59:28.834522 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:28.834487    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" event={"ID":"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6","Type":"ContainerStarted","Data":"de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4"}
Nov 01 12:59:29.593376 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:29.593339    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:59:29.593603 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:29.593578    4095 patch_prober.go:29] interesting pod/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:59:29.593676 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:29.593643    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 12:59:30.490062 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.490017    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" containerID="cri-o://557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136" gracePeriod=45
Nov 01 12:59:30.537723 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.537690    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main" containerID="cri-o://27b0458049f6d9583d6131d12570b9dfc4e3c8cfd78ad5dc2823ea6570a78a91" gracePeriod=45
Nov 01 12:59:30.540983 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.540943    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": dial tcp 10.134.0.68:8000: connect: connection refused"
Nov 01 12:59:30.593610 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.593572    4095 patch_prober.go:29] interesting pod/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:59:30.593733 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.593627    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 12:59:30.842755 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.842366    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:59:30.849718 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.849685    4095 generic.go:356] "Generic (PLEG): container finished" podID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerID="557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136" exitCode=0
Nov 01 12:59:30.849882 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.849728    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"
Nov 01 12:59:30.849882 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.849857    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" event={"ID":"3f047c32-f259-4e7e-b98b-3772fe66aa55","Type":"ContainerDied","Data":"557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136"}
Nov 01 12:59:30.850048 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.849894    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk" event={"ID":"3f047c32-f259-4e7e-b98b-3772fe66aa55","Type":"ContainerDied","Data":"6068e72ae1a243ee36338f3cfddc99680b72d3bc8e03110f63ed70e277da0644"}
Nov 01 12:59:30.850048 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.849913    4095 scope.go:122] "RemoveContainer" containerID="557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136"
Nov 01 12:59:30.851512 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.851491    4095 generic.go:356] "Generic (PLEG): container finished" podID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerID="27b0458049f6d9583d6131d12570b9dfc4e3c8cfd78ad5dc2823ea6570a78a91" exitCode=0
Nov 01 12:59:30.851620 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.851570    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" event={"ID":"73c3bfd2-3551-4ea5-8a5b-98379b29796a","Type":"ContainerDied","Data":"27b0458049f6d9583d6131d12570b9dfc4e3c8cfd78ad5dc2823ea6570a78a91"}
Nov 01 12:59:30.862446 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.862424    4095 scope.go:122] "RemoveContainer" containerID="557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136"
Nov 01 12:59:30.862858 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:59:30.862836    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136\": container with ID starting with 557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136 not found: ID does not exist" containerID="557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136"
Nov 01 12:59:30.862943 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.862869    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136"} err="failed to get container status \"557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136\": rpc error: code = NotFound desc = could not find container \"557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136\": container with ID starting with 557a44ed495846ae15059a078761da04209c09b3549965dc471e22d3e853d136 not found: ID does not exist"
Nov 01 12:59:30.900142 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.900115    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:59:30.983010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.982390    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-dshm\") pod \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") "
Nov 01 12:59:30.983010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.982812    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-home\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-home\") pod \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") "
Nov 01 12:59:30.983010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.982930    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tmp-dir\") pod \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") "
Nov 01 12:59:30.983010 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.982964    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-tmp-dir\") pod \"3f047c32-f259-4e7e-b98b-3772fe66aa55\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") "
Nov 01 12:59:30.983378 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983098    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-home" pod "73c3bfd2-3551-4ea5-8a5b-98379b29796a" (UID: "73c3bfd2-3551-4ea5-8a5b-98379b29796a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:30.983378 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983253    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tmp-dir" pod "73c3bfd2-3551-4ea5-8a5b-98379b29796a" (UID: "73c3bfd2-3551-4ea5-8a5b-98379b29796a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:30.983378 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983360    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73c3bfd2-3551-4ea5-8a5b-98379b29796a-kube-api-access-rxm6x\" (UniqueName: \"kubernetes.io/projected/73c3bfd2-3551-4ea5-8a5b-98379b29796a-kube-api-access-rxm6x\") pod \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") "
Nov 01 12:59:30.983558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983370    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-tmp-dir" pod "3f047c32-f259-4e7e-b98b-3772fe66aa55" (UID: "3f047c32-f259-4e7e-b98b-3772fe66aa55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:30.983558 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983408    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-home\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-home\") pod \"3f047c32-f259-4e7e-b98b-3772fe66aa55\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") "
Nov 01 12:59:30.983663 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983620    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-home" pod "3f047c32-f259-4e7e-b98b-3772fe66aa55" (UID: "3f047c32-f259-4e7e-b98b-3772fe66aa55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:30.983663 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983658    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-model-cache\") pod \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") "
Nov 01 12:59:30.983780 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983727    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-dshm\") pod \"3f047c32-f259-4e7e-b98b-3772fe66aa55\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") "
Nov 01 12:59:30.983780 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983761    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-model-cache\") pod \"3f047c32-f259-4e7e-b98b-3772fe66aa55\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") "
Nov 01 12:59:30.983885 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983798    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f047c32-f259-4e7e-b98b-3772fe66aa55-kube-api-access-9lcr9\" (UniqueName: \"kubernetes.io/projected/3f047c32-f259-4e7e-b98b-3772fe66aa55-kube-api-access-9lcr9\") pod \"3f047c32-f259-4e7e-b98b-3772fe66aa55\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") "
Nov 01 12:59:30.983885 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983832    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f047c32-f259-4e7e-b98b-3772fe66aa55-tls-certs\" (UniqueName: \"kubernetes.io/secret/3f047c32-f259-4e7e-b98b-3772fe66aa55-tls-certs\") pod \"3f047c32-f259-4e7e-b98b-3772fe66aa55\" (UID: \"3f047c32-f259-4e7e-b98b-3772fe66aa55\") "
Nov 01 12:59:30.984063 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983909    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tls-certs\") pod \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\" (UID: \"73c3bfd2-3551-4ea5-8a5b-98379b29796a\") "
Nov 01 12:59:30.984063 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.983914    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-model-cache" pod "73c3bfd2-3551-4ea5-8a5b-98379b29796a" (UID: "73c3bfd2-3551-4ea5-8a5b-98379b29796a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:30.985075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.984584    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:30.985075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.984613    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:30.985075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.984629    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:30.985075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.984645    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:30.985075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.984683    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:30.985075 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.984897    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-model-cache" pod "3f047c32-f259-4e7e-b98b-3772fe66aa55" (UID: "3f047c32-f259-4e7e-b98b-3772fe66aa55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:30.985454 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.985209    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-dshm" pod "73c3bfd2-3551-4ea5-8a5b-98379b29796a" (UID: "73c3bfd2-3551-4ea5-8a5b-98379b29796a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:30.986892 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.986856    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73c3bfd2-3551-4ea5-8a5b-98379b29796a-kube-api-access-rxm6x" pod "73c3bfd2-3551-4ea5-8a5b-98379b29796a" (UID: "73c3bfd2-3551-4ea5-8a5b-98379b29796a"). InnerVolumeSpecName "kube-api-access-rxm6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:30.987153 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.987127    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f047c32-f259-4e7e-b98b-3772fe66aa55-kube-api-access-9lcr9" pod "3f047c32-f259-4e7e-b98b-3772fe66aa55" (UID: "3f047c32-f259-4e7e-b98b-3772fe66aa55"). InnerVolumeSpecName "kube-api-access-9lcr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:30.987786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.987749    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tls-certs" pod "73c3bfd2-3551-4ea5-8a5b-98379b29796a" (UID: "73c3bfd2-3551-4ea5-8a5b-98379b29796a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:59:30.987786 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.987760    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f047c32-f259-4e7e-b98b-3772fe66aa55-tls-certs" pod "3f047c32-f259-4e7e-b98b-3772fe66aa55" (UID: "3f047c32-f259-4e7e-b98b-3772fe66aa55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 12:59:30.988313 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:30.988287    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-dshm" pod "3f047c32-f259-4e7e-b98b-3772fe66aa55" (UID: "3f047c32-f259-4e7e-b98b-3772fe66aa55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 12:59:31.086407 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.086336    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:31.086407 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.086362    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f047c32-f259-4e7e-b98b-3772fe66aa55-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:31.086407 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.086372    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lcr9\" (UniqueName: \"kubernetes.io/projected/3f047c32-f259-4e7e-b98b-3772fe66aa55-kube-api-access-9lcr9\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:31.086407 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.086381    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f047c32-f259-4e7e-b98b-3772fe66aa55-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:31.086407 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.086390    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73c3bfd2-3551-4ea5-8a5b-98379b29796a-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:31.086407 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.086398    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73c3bfd2-3551-4ea5-8a5b-98379b29796a-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:31.086407 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.086408    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxm6x\" (UniqueName: \"kubernetes.io/projected/73c3bfd2-3551-4ea5-8a5b-98379b29796a-kube-api-access-rxm6x\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:31.272181 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.272145    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"]
Nov 01 12:59:31.276869 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.276844    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/distinct-2-w0-94c94b-kserve-fd9756449-88csk"]
Nov 01 12:59:31.593270 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.593229    4095 patch_prober.go:29] interesting pod/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:59:31.593739 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.593296    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 12:59:31.777689 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.777604    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt" podUID="616d14ef-ae4d-40b4-8103-2a42e3613f74" containerName="payload-pre-processing" containerID="cri-o://edaf5cf12d8e86073d5302f2b17ad8b42ea66e1d166f3f69f14e77a4c29897a4" gracePeriod=25
Nov 01 12:59:31.860111 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.860079    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd" event={"ID":"73c3bfd2-3551-4ea5-8a5b-98379b29796a","Type":"ContainerDied","Data":"12a51b66fc12bd3b3cc915f3be4e45cf2ba01588ea1c9a88ac2330371a94d75a"}
Nov 01 12:59:31.860290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.860131    4095 scope.go:122] "RemoveContainer" containerID="27b0458049f6d9583d6131d12570b9dfc4e3c8cfd78ad5dc2823ea6570a78a91"
Nov 01 12:59:31.860290 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:31.860089    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"
Nov 01 12:59:32.030127 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:32.030024    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"]
Nov 01 12:59:32.033813 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:32.033782    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-94c94b/premium-w0-94c94b-kserve-68cf4bb599-k5mvd"]
Nov 01 12:59:32.189618 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:32.189584    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" path="/var/lib/kubelet/pods/3f047c32-f259-4e7e-b98b-3772fe66aa55/volumes"
Nov 01 12:59:32.190156 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:32.190140    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" path="/var/lib/kubelet/pods/73c3bfd2-3551-4ea5-8a5b-98379b29796a/volumes"
Nov 01 12:59:32.593087 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:32.593047    4095 patch_prober.go:29] interesting pod/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:59:32.593291 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:32.593106    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 12:59:33.593216 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:33.593181    4095 patch_prober.go:29] interesting pod/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 01 12:59:33.593385 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:33.593234    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 01 12:59:34.596784 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:34.596752    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:59:34.597267 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:34.597042    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:59:34.597856 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:34.597838    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 12:59:41.631199 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:41.631163    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt" podUID="616d14ef-ae4d-40b4-8103-2a42e3613f74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 12:59:49.112726 ip-10-0-140-168 kubenswrapper[4095]: E1101 12:59:49.112692    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 12:59:51.631344 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:51.631303    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt" podUID="616d14ef-ae4d-40b4-8103-2a42e3613f74" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 01 12:59:51.986924 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:51.986887    4095 generic.go:356] "Generic (PLEG): container finished" podID="616d14ef-ae4d-40b4-8103-2a42e3613f74" containerID="edaf5cf12d8e86073d5302f2b17ad8b42ea66e1d166f3f69f14e77a4c29897a4" exitCode=0
Nov 01 12:59:51.987118 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:51.987036    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt" event={"ID":"616d14ef-ae4d-40b4-8103-2a42e3613f74","Type":"ContainerDied","Data":"edaf5cf12d8e86073d5302f2b17ad8b42ea66e1d166f3f69f14e77a4c29897a4"}
Nov 01 12:59:52.037857 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.037828    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:59:52.180822 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.180717    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/616d14ef-ae4d-40b4-8103-2a42e3613f74-kube-api-access-cchgk\" (UniqueName: \"kubernetes.io/projected/616d14ef-ae4d-40b4-8103-2a42e3613f74-kube-api-access-cchgk\") pod \"616d14ef-ae4d-40b4-8103-2a42e3613f74\" (UID: \"616d14ef-ae4d-40b4-8103-2a42e3613f74\") "
Nov 01 12:59:52.180822 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.180802    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/616d14ef-ae4d-40b4-8103-2a42e3613f74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616d14ef-ae4d-40b4-8103-2a42e3613f74-plugins-config-volume\") pod \"616d14ef-ae4d-40b4-8103-2a42e3613f74\" (UID: \"616d14ef-ae4d-40b4-8103-2a42e3613f74\") "
Nov 01 12:59:52.181651 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.181617    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/616d14ef-ae4d-40b4-8103-2a42e3613f74-plugins-config-volume" pod "616d14ef-ae4d-40b4-8103-2a42e3613f74" (UID: "616d14ef-ae4d-40b4-8103-2a42e3613f74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 12:59:52.183500 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.183472    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/616d14ef-ae4d-40b4-8103-2a42e3613f74-kube-api-access-cchgk" pod "616d14ef-ae4d-40b4-8103-2a42e3613f74" (UID: "616d14ef-ae4d-40b4-8103-2a42e3613f74"). InnerVolumeSpecName "kube-api-access-cchgk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 12:59:52.282848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.282817    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cchgk\" (UniqueName: \"kubernetes.io/projected/616d14ef-ae4d-40b4-8103-2a42e3613f74-kube-api-access-cchgk\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:52.282848 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.282846    4095 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/616d14ef-ae4d-40b4-8103-2a42e3613f74-plugins-config-volume\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 12:59:52.994571 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.994537    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"
Nov 01 12:59:52.995091 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.994565    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt" event={"ID":"616d14ef-ae4d-40b4-8103-2a42e3613f74","Type":"ContainerDied","Data":"7e97c4338208abeeac787dde2c3d5a6776ee615954ed78453c536cb863309d40"}
Nov 01 12:59:52.995091 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:52.994632    4095 scope.go:122] "RemoveContainer" containerID="edaf5cf12d8e86073d5302f2b17ad8b42ea66e1d166f3f69f14e77a4c29897a4"
Nov 01 12:59:53.154667 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:53.154627    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"]
Nov 01 12:59:53.199928 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:53.199898    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-94c94b-6b687668b7-qtrbt"]
Nov 01 12:59:54.225380 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:54.225314    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="616d14ef-ae4d-40b4-8103-2a42e3613f74" path="/var/lib/kubelet/pods/616d14ef-ae4d-40b4-8103-2a42e3613f74/volumes"
Nov 01 12:59:59.499811 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:59.499778    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"]
Nov 01 12:59:59.547567 ip-10-0-140-168 kubenswrapper[4095]: I1101 12:59:59.547531    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"]
Nov 01 13:00:00.205320 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.205277    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"]
Nov 01 13:00:00.205892 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.205875    4095 cpu_manager.go:414] "containerMap: removing container" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy"
Nov 01 13:00:00.205984 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.205893    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy"
Nov 01 13:00:00.205984 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.205916    4095 cpu_manager.go:414] "containerMap: removing container" podUID="616d14ef-ae4d-40b4-8103-2a42e3613f74" containerName="payload-pre-processing"
Nov 01 13:00:00.205984 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.205922    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="616d14ef-ae4d-40b4-8103-2a42e3613f74" containerName="payload-pre-processing"
Nov 01 13:00:00.205984 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.205952    4095 cpu_manager.go:414] "containerMap: removing container" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main"
Nov 01 13:00:00.205984 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.205957    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main"
Nov 01 13:00:00.205984 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.205982    4095 cpu_manager.go:414] "containerMap: removing container" podUID="9d05dbc0-db8f-4084-874e-93b23b734511" containerName="maas-api"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.205988    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d05dbc0-db8f-4084-874e-93b23b734511" containerName="maas-api"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206003    4095 cpu_manager.go:414] "containerMap: removing container" podUID="a9496594-0329-4d04-afc7-7fa862ff4d3b" containerName="payload-processing"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206007    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9496594-0329-4d04-afc7-7fa862ff4d3b" containerName="payload-processing"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206017    4095 cpu_manager.go:414] "containerMap: removing container" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206022    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206109    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="73c3bfd2-3551-4ea5-8a5b-98379b29796a" containerName="main"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206118    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d05dbc0-db8f-4084-874e-93b23b734511" containerName="maas-api"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206127    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9496594-0329-4d04-afc7-7fa862ff4d3b" containerName="payload-processing"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206132    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f047c32-f259-4e7e-b98b-3772fe66aa55" containerName="main"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206138    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="a17bdbf6-1545-43fd-8053-92a61f75edc1" containerName="istio-proxy"
Nov 01 13:00:00.206188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.206144    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="616d14ef-ae4d-40b4-8103-2a42e3613f74" containerName="payload-pre-processing"
Nov 01 13:00:00.210829 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.210807    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:00.226345 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.226315    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"]
Nov 01 13:00:00.246152 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.246116    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:00.267783 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.267751    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"]
Nov 01 13:00:00.282356 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.282325    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"]
Nov 01 13:00:00.285297 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.285265    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb\" (UID: \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:00.285442 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.285321    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ac71af-2277-4c22-9647-a3cca4b0a929-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892300-7djvd\" (UID: \"15ac71af-2277-4c22-9647-a3cca4b0a929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:00.285488 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.285438    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tvw4\" (UniqueName: \"kubernetes.io/projected/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-kube-api-access-7tvw4\") pod \"maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb\" (UID: \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:00.285488 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.285479    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9xms\" (UniqueName: \"kubernetes.io/projected/15ac71af-2277-4c22-9647-a3cca4b0a929-kube-api-access-l9xms\") pod \"maas-api-key-cleanup-29892300-7djvd\" (UID: \"15ac71af-2277-4c22-9647-a3cca4b0a929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:00.322937 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.322908    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:00:00.386389 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.386352    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb\" (UID: \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:00.386590 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.386400    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ac71af-2277-4c22-9647-a3cca4b0a929-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892300-7djvd\" (UID: \"15ac71af-2277-4c22-9647-a3cca4b0a929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:00.386590 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.386426    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tvw4\" (UniqueName: \"kubernetes.io/projected/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-kube-api-access-7tvw4\") pod \"maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb\" (UID: \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:00.386590 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.386445    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9xms\" (UniqueName: \"kubernetes.io/projected/15ac71af-2277-4c22-9647-a3cca4b0a929-kube-api-access-l9xms\") pod \"maas-api-key-cleanup-29892300-7djvd\" (UID: \"15ac71af-2277-4c22-9647-a3cca4b0a929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:00.387180 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.387152    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ac71af-2277-4c22-9647-a3cca4b0a929-maas-api-service-ca\") pod \"maas-api-key-cleanup-29892300-7djvd\" (UID: \"15ac71af-2277-4c22-9647-a3cca4b0a929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:00.387342 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.387280    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb\" (UID: \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:00.444360 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.444326    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9xms\" (UniqueName: \"kubernetes.io/projected/15ac71af-2277-4c22-9647-a3cca4b0a929-kube-api-access-l9xms\") pod \"maas-api-key-cleanup-29892300-7djvd\" (UID: \"15ac71af-2277-4c22-9647-a3cca4b0a929\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:00.444503 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.444371    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tvw4\" (UniqueName: \"kubernetes.io/projected/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-kube-api-access-7tvw4\") pod \"maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb\" (UID: \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:00.529980 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.529903    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:00.570206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.570178    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:00.896176 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:00.896145    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"]
Nov 01 13:00:01.000912 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:01.000887    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"]
Nov 01 13:00:01.001857 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:00:01.001830    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7b40458_1b4d_46f3_ae01_d96dcf1b03aa.slice/crio-1c919c439c4b517de5e16e6c42b15736cb9f5f5bf590799c436e6f2559b93226 WatchSource:0}: Error finding container 1c919c439c4b517de5e16e6c42b15736cb9f5f5bf590799c436e6f2559b93226: Status 404 returned error can't find the container with id 1c919c439c4b517de5e16e6c42b15736cb9f5f5bf590799c436e6f2559b93226
Nov 01 13:00:01.054221 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:01.054185    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" event={"ID":"15ac71af-2277-4c22-9647-a3cca4b0a929","Type":"ContainerStarted","Data":"39861090b303acb37738672568b687a7c9e5e5f3a13c449b4358d2f756baa3a1"}
Nov 01 13:00:01.055458 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:01.055433    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb" event={"ID":"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa","Type":"ContainerStarted","Data":"1c919c439c4b517de5e16e6c42b15736cb9f5f5bf590799c436e6f2559b93226"}
Nov 01 13:00:02.392723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.392686    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"]
Nov 01 13:00:02.393234 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.392914    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54" podUID="fff1b7f2-ba38-4aab-94f8-00c8f23ef154" containerName="maas-api" containerID="cri-o://ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb" gracePeriod=30
Nov 01 13:00:02.694868 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.694831    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 13:00:02.719580 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.719545    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-metrics-certs\") pod \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") "
Nov 01 13:00:02.719759 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.719595    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-kube-api-access-qn582\" (UniqueName: \"kubernetes.io/projected/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-kube-api-access-qn582\") pod \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") "
Nov 01 13:00:02.719759 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.719647    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls\") pod \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") "
Nov 01 13:00:02.719759 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.719703    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-openshift-service-ca\") pod \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\" (UID: \"fff1b7f2-ba38-4aab-94f8-00c8f23ef154\") "
Nov 01 13:00:02.720165 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.720134    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-openshift-service-ca" pod "fff1b7f2-ba38-4aab-94f8-00c8f23ef154" (UID: "fff1b7f2-ba38-4aab-94f8-00c8f23ef154"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:02.721832 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.721809    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls" pod "fff1b7f2-ba38-4aab-94f8-00c8f23ef154" (UID: "fff1b7f2-ba38-4aab-94f8-00c8f23ef154"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:02.722249 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.722221    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-kube-api-access-qn582" pod "fff1b7f2-ba38-4aab-94f8-00c8f23ef154" (UID: "fff1b7f2-ba38-4aab-94f8-00c8f23ef154"). InnerVolumeSpecName "kube-api-access-qn582". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:02.722249 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.722221    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-metrics-certs" pod "fff1b7f2-ba38-4aab-94f8-00c8f23ef154" (UID: "fff1b7f2-ba38-4aab-94f8-00c8f23ef154"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:02.821142 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.821114    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:02.821142 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.821139    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn582\" (UniqueName: \"kubernetes.io/projected/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-kube-api-access-qn582\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:02.821142 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.821149    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:02.821359 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:02.821158    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fff1b7f2-ba38-4aab-94f8-00c8f23ef154-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:03.069837 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:03.069753    4095 generic.go:356] "Generic (PLEG): container finished" podID="fff1b7f2-ba38-4aab-94f8-00c8f23ef154" containerID="ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb" exitCode=0
Nov 01 13:00:03.069837 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:03.069791    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54" event={"ID":"fff1b7f2-ba38-4aab-94f8-00c8f23ef154","Type":"ContainerDied","Data":"ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb"}
Nov 01 13:00:03.069837 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:03.069801    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"
Nov 01 13:00:03.069837 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:03.069835    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54" event={"ID":"fff1b7f2-ba38-4aab-94f8-00c8f23ef154","Type":"ContainerDied","Data":"6464c9c32120174a2e8b502f63791e19530fc045572559fa6e02f831365b3860"}
Nov 01 13:00:03.070166 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:03.069859    4095 scope.go:122] "RemoveContainer" containerID="ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb"
Nov 01 13:00:03.080854 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:03.080838    4095 scope.go:122] "RemoveContainer" containerID="ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb"
Nov 01 13:00:03.081125 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:03.081105    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb\": container with ID starting with ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb not found: ID does not exist" containerID="ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb"
Nov 01 13:00:03.081172 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:03.081133    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb"} err="failed to get container status \"ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb\": rpc error: code = NotFound desc = could not find container \"ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb\": container with ID starting with ea7c26294fa9b47ad7f9876f3daf47ed4b2339d3f65c01a32ad71c6314d9c8bb not found: ID does not exist"
Nov 01 13:00:03.229751 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:03.229718    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"]
Nov 01 13:00:03.235633 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:03.235608    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-78c69d-5cc799fdf8-pbz54"]
Nov 01 13:00:04.201105 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:04.201069    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fff1b7f2-ba38-4aab-94f8-00c8f23ef154" path="/var/lib/kubelet/pods/fff1b7f2-ba38-4aab-94f8-00c8f23ef154/volumes"
Nov 01 13:00:06.099170 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:06.099131    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" event={"ID":"15ac71af-2277-4c22-9647-a3cca4b0a929","Type":"ContainerStarted","Data":"f992e54fdb31102a3cd9b062a2028cf8a58fe17aecac84a44b207c08f2c4281a"}
Nov 01 13:00:06.208952 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:06.208854    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" podStartSLOduration=1.422170663 podStartE2EDuration="6.208839228s" totalImagesPullingTime="4.786668565s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:06.207332332 +0000 UTC m=+2266.125128000" watchObservedRunningTime="2026-11-01 13:00:06.208839228 +0000 UTC m=+2266.126634894"
Nov 01 13:00:07.106713 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:07.106678    4095 generic.go:356] "Generic (PLEG): container finished" podID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerID="c3942d981fb5774447fbffd3fbfa10e7bf2e636befbab1769890e40cb3b67bc0" exitCode=60
Nov 01 13:00:07.107234 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:07.106741    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb" event={"ID":"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa","Type":"ContainerDied","Data":"c3942d981fb5774447fbffd3fbfa10e7bf2e636befbab1769890e40cb3b67bc0"}
Nov 01 13:00:07.107234 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:07.107075    4095 scope.go:122] "RemoveContainer" containerID="c3942d981fb5774447fbffd3fbfa10e7bf2e636befbab1769890e40cb3b67bc0"
Nov 01 13:00:07.108626 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:07.108602    4095 generic.go:356] "Generic (PLEG): container finished" podID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerID="f992e54fdb31102a3cd9b062a2028cf8a58fe17aecac84a44b207c08f2c4281a" exitCode=60
Nov 01 13:00:07.108728 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:07.108698    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" event={"ID":"15ac71af-2277-4c22-9647-a3cca4b0a929","Type":"ContainerDied","Data":"f992e54fdb31102a3cd9b062a2028cf8a58fe17aecac84a44b207c08f2c4281a"}
Nov 01 13:00:07.108900 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:07.108884    4095 scope.go:122] "RemoveContainer" containerID="f992e54fdb31102a3cd9b062a2028cf8a58fe17aecac84a44b207c08f2c4281a"
Nov 01 13:00:08.117890 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.117854    4095 generic.go:356] "Generic (PLEG): container finished" podID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerID="d5c5864e6f9e360b21e0ff66eb69295be4f72cbfc482aba0c7576b2ecf1fbab6" exitCode=60
Nov 01 13:00:08.118372 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.117929    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" event={"ID":"15ac71af-2277-4c22-9647-a3cca4b0a929","Type":"ContainerDied","Data":"d5c5864e6f9e360b21e0ff66eb69295be4f72cbfc482aba0c7576b2ecf1fbab6"}
Nov 01 13:00:08.118372 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.118013    4095 scope.go:122] "RemoveContainer" containerID="f992e54fdb31102a3cd9b062a2028cf8a58fe17aecac84a44b207c08f2c4281a"
Nov 01 13:00:08.118372 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.118254    4095 scope.go:122] "RemoveContainer" containerID="d5c5864e6f9e360b21e0ff66eb69295be4f72cbfc482aba0c7576b2ecf1fbab6"
Nov 01 13:00:08.118571 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:08.118549    4095 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892300-7djvd_odh-ai-gateway-infra(15ac71af-2277-4c22-9647-a3cca4b0a929)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929"
Nov 01 13:00:08.120145 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.120124    4095 generic.go:356] "Generic (PLEG): container finished" podID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerID="56366978c0a39e0d0027aa43cffce2481c09617e33c5ab46b7f11fd5d35eb9ea" exitCode=60
Nov 01 13:00:08.120242 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.120203    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb" event={"ID":"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa","Type":"ContainerDied","Data":"56366978c0a39e0d0027aa43cffce2481c09617e33c5ab46b7f11fd5d35eb9ea"}
Nov 01 13:00:08.120443 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.120429    4095 scope.go:122] "RemoveContainer" containerID="56366978c0a39e0d0027aa43cffce2481c09617e33c5ab46b7f11fd5d35eb9ea"
Nov 01 13:00:08.120672 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:08.120657    4095 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb_odh-ai-gateway-infra(f7b40458-1b4d-46f3-ae01-d96dcf1b03aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa"
Nov 01 13:00:08.130564 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.130547    4095 scope.go:122] "RemoveContainer" containerID="c3942d981fb5774447fbffd3fbfa10e7bf2e636befbab1769890e40cb3b67bc0"
Nov 01 13:00:08.360126 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.360098    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"]
Nov 01 13:00:08.360326 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:08.360302    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy" containerID="cri-o://17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085" gracePeriod=30
Nov 01 13:00:09.128394 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:09.128363    4095 scope.go:122] "RemoveContainer" containerID="d5c5864e6f9e360b21e0ff66eb69295be4f72cbfc482aba0c7576b2ecf1fbab6"
Nov 01 13:00:09.128875 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:09.128711    4095 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29892300-7djvd_odh-ai-gateway-infra(15ac71af-2277-4c22-9647-a3cca4b0a929)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929"
Nov 01 13:00:09.129902 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:09.129882    4095 scope.go:122] "RemoveContainer" containerID="56366978c0a39e0d0027aa43cffce2481c09617e33c5ab46b7f11fd5d35eb9ea"
Nov 01 13:00:09.130151 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:09.130134    4095 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb_odh-ai-gateway-infra(f7b40458-1b4d-46f3-ae01-d96dcf1b03aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa"
Nov 01 13:00:12.474037 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:12.473959    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"]
Nov 01 13:00:12.474519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:12.474218    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy" containerID="cri-o://6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68" gracePeriod=30
Nov 01 13:00:13.687369 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.687327    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 13:00:13.788711 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.788677    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-certs\") pod \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") "
Nov 01 13:00:13.788903 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.788792    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-podinfo\") pod \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") "
Nov 01 13:00:13.788903 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.788830    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-data\") pod \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") "
Nov 01 13:00:13.788903 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.788857    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-token\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-token\") pod \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") "
Nov 01 13:00:13.789109 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.788926    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istiod-ca-cert\") pod \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") "
Nov 01 13:00:13.789109 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.788956    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-certs" pod "44a51e92-f7f8-4b70-82ac-5439e2fc347f" (UID: "44a51e92-f7f8-4b70-82ac-5439e2fc347f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:13.789109 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789032    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-credential-socket\") pod \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") "
Nov 01 13:00:13.789109 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789102    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-kube-api-access-7z5kc\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-kube-api-access-7z5kc\") pod \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") "
Nov 01 13:00:13.789302 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789111    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-data" pod "44a51e92-f7f8-4b70-82ac-5439e2fc347f" (UID: "44a51e92-f7f8-4b70-82ac-5439e2fc347f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:13.789302 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789164    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-envoy\") pod \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") "
Nov 01 13:00:13.789302 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789237    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-socket\") pod \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\" (UID: \"44a51e92-f7f8-4b70-82ac-5439e2fc347f\") "
Nov 01 13:00:13.789302 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789287    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-credential-socket" pod "44a51e92-f7f8-4b70-82ac-5439e2fc347f" (UID: "44a51e92-f7f8-4b70-82ac-5439e2fc347f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:13.789486 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789355    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istiod-ca-cert" pod "44a51e92-f7f8-4b70-82ac-5439e2fc347f" (UID: "44a51e92-f7f8-4b70-82ac-5439e2fc347f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:13.789534 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789478    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-socket" pod "44a51e92-f7f8-4b70-82ac-5439e2fc347f" (UID: "44a51e92-f7f8-4b70-82ac-5439e2fc347f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:13.789835 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789814    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:13.789929 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789840    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:13.789929 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789856    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:13.789929 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789872    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:13.789929 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.789887    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:13.791168 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.791138    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-token" pod "44a51e92-f7f8-4b70-82ac-5439e2fc347f" (UID: "44a51e92-f7f8-4b70-82ac-5439e2fc347f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:13.791266 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.791194    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-podinfo" pod "44a51e92-f7f8-4b70-82ac-5439e2fc347f" (UID: "44a51e92-f7f8-4b70-82ac-5439e2fc347f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:00:13.791308 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.791289    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-envoy" pod "44a51e92-f7f8-4b70-82ac-5439e2fc347f" (UID: "44a51e92-f7f8-4b70-82ac-5439e2fc347f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:13.791666 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.791638    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-kube-api-access-7z5kc" pod "44a51e92-f7f8-4b70-82ac-5439e2fc347f" (UID: "44a51e92-f7f8-4b70-82ac-5439e2fc347f"). InnerVolumeSpecName "kube-api-access-7z5kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:13.890703 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.890676    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:13.890703 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.890701    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z5kc\" (UniqueName: \"kubernetes.io/projected/44a51e92-f7f8-4b70-82ac-5439e2fc347f-kube-api-access-7z5kc\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:13.890703 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.890712    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:13.890922 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:13.890728    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/44a51e92-f7f8-4b70-82ac-5439e2fc347f-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:14.159981 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.159929    4095 generic.go:356] "Generic (PLEG): container finished" podID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerID="17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085" exitCode=0
Nov 01 13:00:14.160154 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.160011    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"
Nov 01 13:00:14.160154 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.160025    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" event={"ID":"44a51e92-f7f8-4b70-82ac-5439e2fc347f","Type":"ContainerDied","Data":"17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085"}
Nov 01 13:00:14.160154 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.160089    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc" event={"ID":"44a51e92-f7f8-4b70-82ac-5439e2fc347f","Type":"ContainerDied","Data":"a4b901cfdd806e1da9fe0e2197dc5da28beb735ab4b88f364d5cf3b810a3d664"}
Nov 01 13:00:14.160154 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.160113    4095 scope.go:122] "RemoveContainer" containerID="17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085"
Nov 01 13:00:14.170366 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.170349    4095 scope.go:122] "RemoveContainer" containerID="17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085"
Nov 01 13:00:14.170635 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:14.170619    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085\": container with ID starting with 17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085 not found: ID does not exist" containerID="17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085"
Nov 01 13:00:14.170681 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.170643    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085"} err="failed to get container status \"17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085\": rpc error: code = NotFound desc = could not find container \"17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085\": container with ID starting with 17eab7c66a6238b2dd07ad9056043b6c295a75f57be0b4944ae0d418c289a085 not found: ID does not exist"
Nov 01 13:00:14.317089 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.317051    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"]
Nov 01 13:00:14.320531 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.320507    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-j62wc"]
Nov 01 13:00:14.521158 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.521056    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" containerID="cri-o://863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5" gracePeriod=45
Nov 01 13:00:14.528775 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.528745    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main" probeResult="failure" output="Get \"https://10.134.0.65:8000/ready\": dial tcp 10.134.0.65:8000: connect: connection refused"
Nov 01 13:00:14.568126 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.568090    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main" containerID="cri-o://52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0" gracePeriod=45
Nov 01 13:00:14.889902 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.889780    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 13:00:14.892826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.892800    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 13:00:14.902259 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902234    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-tmp-dir\") pod \"9d5c4201-96c1-40b9-be00-774ccc27e376\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") "
Nov 01 13:00:14.902395 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902270    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-model-cache\") pod \"9d5c4201-96c1-40b9-be00-774ccc27e376\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") "
Nov 01 13:00:14.902395 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902367    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5c4201-96c1-40b9-be00-774ccc27e376-kube-api-access-gstpw\" (UniqueName: \"kubernetes.io/projected/9d5c4201-96c1-40b9-be00-774ccc27e376-kube-api-access-gstpw\") pod \"9d5c4201-96c1-40b9-be00-774ccc27e376\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") "
Nov 01 13:00:14.902526 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902401    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-tmp-dir\") pod \"c70639c6-8601-4a9c-866c-d194156300b6\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") "
Nov 01 13:00:14.902526 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902432    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-dshm\") pod \"9d5c4201-96c1-40b9-be00-774ccc27e376\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") "
Nov 01 13:00:14.902526 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902470    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5c4201-96c1-40b9-be00-774ccc27e376-tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5c4201-96c1-40b9-be00-774ccc27e376-tls-certs\") pod \"9d5c4201-96c1-40b9-be00-774ccc27e376\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") "
Nov 01 13:00:14.902687 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902530    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-model-cache\") pod \"c70639c6-8601-4a9c-866c-d194156300b6\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") "
Nov 01 13:00:14.902687 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902575    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c70639c6-8601-4a9c-866c-d194156300b6-tls-certs\" (UniqueName: \"kubernetes.io/secret/c70639c6-8601-4a9c-866c-d194156300b6-tls-certs\") pod \"c70639c6-8601-4a9c-866c-d194156300b6\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") "
Nov 01 13:00:14.902687 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902672    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c70639c6-8601-4a9c-866c-d194156300b6-kube-api-access-69v5x\" (UniqueName: \"kubernetes.io/projected/c70639c6-8601-4a9c-866c-d194156300b6-kube-api-access-69v5x\") pod \"c70639c6-8601-4a9c-866c-d194156300b6\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") "
Nov 01 13:00:14.902849 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902710    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-home\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-home\") pod \"9d5c4201-96c1-40b9-be00-774ccc27e376\" (UID: \"9d5c4201-96c1-40b9-be00-774ccc27e376\") "
Nov 01 13:00:14.902849 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902743    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-dshm\") pod \"c70639c6-8601-4a9c-866c-d194156300b6\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") "
Nov 01 13:00:14.902849 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.902822    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-home\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-home\") pod \"c70639c6-8601-4a9c-866c-d194156300b6\" (UID: \"c70639c6-8601-4a9c-866c-d194156300b6\") "
Nov 01 13:00:14.904200 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.903070    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-tmp-dir" pod "9d5c4201-96c1-40b9-be00-774ccc27e376" (UID: "9d5c4201-96c1-40b9-be00-774ccc27e376"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:14.904200 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.903290    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-model-cache" pod "c70639c6-8601-4a9c-866c-d194156300b6" (UID: "c70639c6-8601-4a9c-866c-d194156300b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:14.904200 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.903483    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:14.904200 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.903503    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:14.904200 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.903748    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-tmp-dir" pod "c70639c6-8601-4a9c-866c-d194156300b6" (UID: "c70639c6-8601-4a9c-866c-d194156300b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:14.904200 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.903937    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-home" pod "9d5c4201-96c1-40b9-be00-774ccc27e376" (UID: "9d5c4201-96c1-40b9-be00-774ccc27e376"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:14.904200 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.903951    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-model-cache" pod "9d5c4201-96c1-40b9-be00-774ccc27e376" (UID: "9d5c4201-96c1-40b9-be00-774ccc27e376"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:14.904670 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.904507    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-home" pod "c70639c6-8601-4a9c-866c-d194156300b6" (UID: "c70639c6-8601-4a9c-866c-d194156300b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:14.908469 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.908389    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c70639c6-8601-4a9c-866c-d194156300b6-tls-certs" pod "c70639c6-8601-4a9c-866c-d194156300b6" (UID: "c70639c6-8601-4a9c-866c-d194156300b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:14.909106 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.909067    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5c4201-96c1-40b9-be00-774ccc27e376-kube-api-access-gstpw" pod "9d5c4201-96c1-40b9-be00-774ccc27e376" (UID: "9d5c4201-96c1-40b9-be00-774ccc27e376"). InnerVolumeSpecName "kube-api-access-gstpw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:14.909329 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.909234    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-dshm" pod "c70639c6-8601-4a9c-866c-d194156300b6" (UID: "c70639c6-8601-4a9c-866c-d194156300b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:14.909329 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.909298    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70639c6-8601-4a9c-866c-d194156300b6-kube-api-access-69v5x" pod "c70639c6-8601-4a9c-866c-d194156300b6" (UID: "c70639c6-8601-4a9c-866c-d194156300b6"). InnerVolumeSpecName "kube-api-access-69v5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:14.910056 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.910037    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-dshm" pod "9d5c4201-96c1-40b9-be00-774ccc27e376" (UID: "9d5c4201-96c1-40b9-be00-774ccc27e376"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:14.911817 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:14.911761    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5c4201-96c1-40b9-be00-774ccc27e376-tls-certs" pod "9d5c4201-96c1-40b9-be00-774ccc27e376" (UID: "9d5c4201-96c1-40b9-be00-774ccc27e376"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:15.004649 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004613    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69v5x\" (UniqueName: \"kubernetes.io/projected/c70639c6-8601-4a9c-866c-d194156300b6-kube-api-access-69v5x\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.004649 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004644    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.004885 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004660    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.004885 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004672    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.004885 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004680    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.004885 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004690    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gstpw\" (UniqueName: \"kubernetes.io/projected/9d5c4201-96c1-40b9-be00-774ccc27e376-kube-api-access-gstpw\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.004885 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004699    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c70639c6-8601-4a9c-866c-d194156300b6-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.004885 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004706    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d5c4201-96c1-40b9-be00-774ccc27e376-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.004885 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004714    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d5c4201-96c1-40b9-be00-774ccc27e376-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.004885 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.004721    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c70639c6-8601-4a9c-866c-d194156300b6-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:15.168409 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.168371    4095 generic.go:356] "Generic (PLEG): container finished" podID="c70639c6-8601-4a9c-866c-d194156300b6" containerID="863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5" exitCode=0
Nov 01 13:00:15.168409 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.168410    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"
Nov 01 13:00:15.168639 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.168450    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" event={"ID":"c70639c6-8601-4a9c-866c-d194156300b6","Type":"ContainerDied","Data":"863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5"}
Nov 01 13:00:15.168639 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.168497    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw" event={"ID":"c70639c6-8601-4a9c-866c-d194156300b6","Type":"ContainerDied","Data":"cc20a95e6824795288fc03ecc7720509ec5205b634b779caa46f6301e0a63654"}
Nov 01 13:00:15.168639 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.168519    4095 scope.go:122] "RemoveContainer" containerID="863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5"
Nov 01 13:00:15.171150 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.171122    4095 generic.go:356] "Generic (PLEG): container finished" podID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerID="52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0" exitCode=0
Nov 01 13:00:15.171263 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.171169    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"
Nov 01 13:00:15.171263 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.171212    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" event={"ID":"9d5c4201-96c1-40b9-be00-774ccc27e376","Type":"ContainerDied","Data":"52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0"}
Nov 01 13:00:15.171263 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.171260    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g" event={"ID":"9d5c4201-96c1-40b9-be00-774ccc27e376","Type":"ContainerDied","Data":"8f7eb31f493fef65b602a4c5d0008e3d3ccb6b0eba674eb9fbdd172d7ec98a03"}
Nov 01 13:00:15.179723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.179685    4095 scope.go:122] "RemoveContainer" containerID="863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5"
Nov 01 13:00:15.179964 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:15.179941    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5\": container with ID starting with 863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5 not found: ID does not exist" containerID="863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5"
Nov 01 13:00:15.180077 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.179991    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5"} err="failed to get container status \"863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5\": rpc error: code = NotFound desc = could not find container \"863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5\": container with ID starting with 863e97067e95f3a93c6812e259da35510ed49f08cb116ec9f1ddcaabf87eb1d5 not found: ID does not exist"
Nov 01 13:00:15.180077 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.180016    4095 scope.go:122] "RemoveContainer" containerID="52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0"
Nov 01 13:00:15.189715 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.189698    4095 scope.go:122] "RemoveContainer" containerID="52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0"
Nov 01 13:00:15.190019 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:15.189965    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0\": container with ID starting with 52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0 not found: ID does not exist" containerID="52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0"
Nov 01 13:00:15.190074 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.190030    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0"} err="failed to get container status \"52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0\": rpc error: code = NotFound desc = could not find container \"52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0\": container with ID starting with 52304dec9206dff7030b221804d2bf272e6317e2d40cfe017bd15d49c29b81b0 not found: ID does not exist"
Nov 01 13:00:15.342838 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.342782    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"]
Nov 01 13:00:15.355509 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.355480    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-w1-5b1bd4-kserve-6966db5cd8-lmw6g"]
Nov 01 13:00:15.460679 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.460603    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"]
Nov 01 13:00:15.462884 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:15.462863    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-5b1bd4/distinct-2-w1-5b1bd4-kserve-6c55d8c9dc-lhgxw"]
Nov 01 13:00:16.188312 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:16.188278    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" path="/var/lib/kubelet/pods/44a51e92-f7f8-4b70-82ac-5439e2fc347f/volumes"
Nov 01 13:00:16.188797 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:16.188782    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" path="/var/lib/kubelet/pods/9d5c4201-96c1-40b9-be00-774ccc27e376/volumes"
Nov 01 13:00:16.189270 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:16.189255    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c70639c6-8601-4a9c-866c-d194156300b6" path="/var/lib/kubelet/pods/c70639c6-8601-4a9c-866c-d194156300b6/volumes"
Nov 01 13:00:17.740548 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.740497    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 13:00:17.836531 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836495    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-token\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-token\") pod \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") "
Nov 01 13:00:17.836673 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836548    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-kube-api-access-45zgl\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-kube-api-access-45zgl\") pod \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") "
Nov 01 13:00:17.836673 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836615    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-certs\") pod \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") "
Nov 01 13:00:17.836673 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836667    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-data\") pod \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") "
Nov 01 13:00:17.836848 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836781    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-podinfo\") pod \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") "
Nov 01 13:00:17.836848 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836821    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-credential-socket\") pod \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") "
Nov 01 13:00:17.836944 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836858    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-socket\") pod \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") "
Nov 01 13:00:17.836944 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836881    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-certs" pod "4dc78e78-3863-44a7-b0b1-5f96566e8ec8" (UID: "4dc78e78-3863-44a7-b0b1-5f96566e8ec8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:17.836944 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836898    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istiod-ca-cert\") pod \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") "
Nov 01 13:00:17.836944 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836908    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-data" pod "4dc78e78-3863-44a7-b0b1-5f96566e8ec8" (UID: "4dc78e78-3863-44a7-b0b1-5f96566e8ec8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:17.837175 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.836943    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-envoy\") pod \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\" (UID: \"4dc78e78-3863-44a7-b0b1-5f96566e8ec8\") "
Nov 01 13:00:17.837175 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.837093    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-credential-socket" pod "4dc78e78-3863-44a7-b0b1-5f96566e8ec8" (UID: "4dc78e78-3863-44a7-b0b1-5f96566e8ec8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:17.837175 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.837167    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-socket" pod "4dc78e78-3863-44a7-b0b1-5f96566e8ec8" (UID: "4dc78e78-3863-44a7-b0b1-5f96566e8ec8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:17.837311 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.837282    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istiod-ca-cert" pod "4dc78e78-3863-44a7-b0b1-5f96566e8ec8" (UID: "4dc78e78-3863-44a7-b0b1-5f96566e8ec8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:17.837546 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.837523    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:17.837651 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.837550    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:17.837651 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.837567    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:17.837651 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.837582    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:17.837651 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.837598    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:17.838921 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.838896    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-token" pod "4dc78e78-3863-44a7-b0b1-5f96566e8ec8" (UID: "4dc78e78-3863-44a7-b0b1-5f96566e8ec8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:17.839035 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.838942    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-podinfo" pod "4dc78e78-3863-44a7-b0b1-5f96566e8ec8" (UID: "4dc78e78-3863-44a7-b0b1-5f96566e8ec8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:00:17.839088 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.839052    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-envoy" pod "4dc78e78-3863-44a7-b0b1-5f96566e8ec8" (UID: "4dc78e78-3863-44a7-b0b1-5f96566e8ec8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:17.839246 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.839230    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-kube-api-access-45zgl" pod "4dc78e78-3863-44a7-b0b1-5f96566e8ec8" (UID: "4dc78e78-3863-44a7-b0b1-5f96566e8ec8"). InnerVolumeSpecName "kube-api-access-45zgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:17.938891 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.938863    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:17.938891 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.938889    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:17.939119 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.938900    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:17.939119 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:17.938909    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45zgl\" (UniqueName: \"kubernetes.io/projected/4dc78e78-3863-44a7-b0b1-5f96566e8ec8-kube-api-access-45zgl\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:18.194120 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.194088    4095 generic.go:356] "Generic (PLEG): container finished" podID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerID="6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68" exitCode=0
Nov 01 13:00:18.194289 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.194142    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"
Nov 01 13:00:18.194289 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.194162    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" event={"ID":"4dc78e78-3863-44a7-b0b1-5f96566e8ec8","Type":"ContainerDied","Data":"6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68"}
Nov 01 13:00:18.194289 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.194208    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs" event={"ID":"4dc78e78-3863-44a7-b0b1-5f96566e8ec8","Type":"ContainerDied","Data":"7806faa5cb63f2e2982dc80e862137dc5b925df3ab995f0f4dc7115492a4c4a5"}
Nov 01 13:00:18.194289 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.194237    4095 scope.go:122] "RemoveContainer" containerID="6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68"
Nov 01 13:00:18.207844 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.207821    4095 scope.go:122] "RemoveContainer" containerID="caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc"
Nov 01 13:00:18.216914 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.216896    4095 scope.go:122] "RemoveContainer" containerID="6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68"
Nov 01 13:00:18.217201 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:18.217181    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68\": container with ID starting with 6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68 not found: ID does not exist" containerID="6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68"
Nov 01 13:00:18.217298 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.217212    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68"} err="failed to get container status \"6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68\": rpc error: code = NotFound desc = could not find container \"6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68\": container with ID starting with 6f1c8d91789181380d644189e1feae93d170f163a7e41f60526edf847a022f68 not found: ID does not exist"
Nov 01 13:00:18.217298 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.217236    4095 scope.go:122] "RemoveContainer" containerID="caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc"
Nov 01 13:00:18.217474 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:18.217457    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc\": container with ID starting with caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc not found: ID does not exist" containerID="caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc"
Nov 01 13:00:18.217530 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.217484    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc"} err="failed to get container status \"caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc\": rpc error: code = NotFound desc = could not find container \"caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc\": container with ID starting with caf6feb754d04d1f2659986fd2702f0bde983d0d5cdcfaf70c8445c6123b49dc not found: ID does not exist"
Nov 01 13:00:18.351886 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.351860    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"]
Nov 01 13:00:18.399317 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:18.399280    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-78c69d-openshift-default-865f5979c5-5grbs"]
Nov 01 13:00:19.114005 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:19.113937    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:00:20.189745 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:20.189713    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" path="/var/lib/kubelet/pods/4dc78e78-3863-44a7-b0b1-5f96566e8ec8/volumes"
Nov 01 13:00:21.182944 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:21.182913    4095 scope.go:122] "RemoveContainer" containerID="56366978c0a39e0d0027aa43cffce2481c09617e33c5ab46b7f11fd5d35eb9ea"
Nov 01 13:00:22.182164 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:22.182133    4095 scope.go:122] "RemoveContainer" containerID="d5c5864e6f9e360b21e0ff66eb69295be4f72cbfc482aba0c7576b2ecf1fbab6"
Nov 01 13:00:22.219619 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:22.219580    4095 generic.go:356] "Generic (PLEG): container finished" podID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerID="d8d31d220464cd28e7e2de1ee55147d45bb68d1f2069d7fed15db761f2247e67" exitCode=60
Nov 01 13:00:22.219743 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:22.219669    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb" event={"ID":"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa","Type":"ContainerDied","Data":"d8d31d220464cd28e7e2de1ee55147d45bb68d1f2069d7fed15db761f2247e67"}
Nov 01 13:00:22.219782 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:22.219742    4095 scope.go:122] "RemoveContainer" containerID="56366978c0a39e0d0027aa43cffce2481c09617e33c5ab46b7f11fd5d35eb9ea"
Nov 01 13:00:22.220169 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:22.220151    4095 scope.go:122] "RemoveContainer" containerID="d8d31d220464cd28e7e2de1ee55147d45bb68d1f2069d7fed15db761f2247e67"
Nov 01 13:00:22.220472 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:22.220449    4095 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb_odh-ai-gateway-infra(f7b40458-1b4d-46f3-ae01-d96dcf1b03aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa"
Nov 01 13:00:23.228915 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.228879    4095 generic.go:356] "Generic (PLEG): container finished" podID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerID="05213d8ce16f22d7987edc8a89cac1af0e6fc912203804c46bd97ef29d7c655d" exitCode=60
Nov 01 13:00:23.229426 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.228929    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" event={"ID":"15ac71af-2277-4c22-9647-a3cca4b0a929","Type":"ContainerDied","Data":"05213d8ce16f22d7987edc8a89cac1af0e6fc912203804c46bd97ef29d7c655d"}
Nov 01 13:00:23.229426 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.228996    4095 scope.go:122] "RemoveContainer" containerID="d5c5864e6f9e360b21e0ff66eb69295be4f72cbfc482aba0c7576b2ecf1fbab6"
Nov 01 13:00:23.229538 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.229440    4095 scope.go:122] "RemoveContainer" containerID="05213d8ce16f22d7987edc8a89cac1af0e6fc912203804c46bd97ef29d7c655d"
Nov 01 13:00:23.229761 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:23.229735    4095 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29892300-7djvd_odh-ai-gateway-infra(15ac71af-2277-4c22-9647-a3cca4b0a929)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929"
Nov 01 13:00:23.338029 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.337937    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"]
Nov 01 13:00:23.489019 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.488954    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:23.605543 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.605458    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-kube-api-access-7tvw4\" (UniqueName: \"kubernetes.io/projected/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-kube-api-access-7tvw4\") pod \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\" (UID: \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\") "
Nov 01 13:00:23.605704 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.605542    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-maas-api-service-ca\") pod \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\" (UID: \"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa\") "
Nov 01 13:00:23.606290 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.606254    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-maas-api-service-ca" pod "f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" (UID: "f7b40458-1b4d-46f3-ae01-d96dcf1b03aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:23.607759 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.607735    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-kube-api-access-7tvw4" pod "f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" (UID: "f7b40458-1b4d-46f3-ae01-d96dcf1b03aa"). InnerVolumeSpecName "kube-api-access-7tvw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:23.707101 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.707064    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tvw4\" (UniqueName: \"kubernetes.io/projected/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-kube-api-access-7tvw4\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:23.707101 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:23.707096    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa-maas-api-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:24.235832 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.235798    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"
Nov 01 13:00:24.235832 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.235816    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb" event={"ID":"f7b40458-1b4d-46f3-ae01-d96dcf1b03aa","Type":"ContainerDied","Data":"1c919c439c4b517de5e16e6c42b15736cb9f5f5bf590799c436e6f2559b93226"}
Nov 01 13:00:24.236414 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.235897    4095 scope.go:122] "RemoveContainer" containerID="d8d31d220464cd28e7e2de1ee55147d45bb68d1f2069d7fed15db761f2247e67"
Nov 01 13:00:24.348950 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.348918    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"]
Nov 01 13:00:24.391538 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.391474    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"]
Nov 01 13:00:24.396181 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.396153    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-2810c2-29892300-b8nsb"]
Nov 01 13:00:24.508646 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.508618    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:24.618235 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.618202    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15ac71af-2277-4c22-9647-a3cca4b0a929-kube-api-access-l9xms\" (UniqueName: \"kubernetes.io/projected/15ac71af-2277-4c22-9647-a3cca4b0a929-kube-api-access-l9xms\") pod \"15ac71af-2277-4c22-9647-a3cca4b0a929\" (UID: \"15ac71af-2277-4c22-9647-a3cca4b0a929\") "
Nov 01 13:00:24.618533 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.618511    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15ac71af-2277-4c22-9647-a3cca4b0a929-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ac71af-2277-4c22-9647-a3cca4b0a929-maas-api-service-ca\") pod \"15ac71af-2277-4c22-9647-a3cca4b0a929\" (UID: \"15ac71af-2277-4c22-9647-a3cca4b0a929\") "
Nov 01 13:00:24.619095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.619065    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15ac71af-2277-4c22-9647-a3cca4b0a929-maas-api-service-ca" pod "15ac71af-2277-4c22-9647-a3cca4b0a929" (UID: "15ac71af-2277-4c22-9647-a3cca4b0a929"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:24.619302 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.619277    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/15ac71af-2277-4c22-9647-a3cca4b0a929-maas-api-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:24.621927 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.621596    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15ac71af-2277-4c22-9647-a3cca4b0a929-kube-api-access-l9xms" pod "15ac71af-2277-4c22-9647-a3cca4b0a929" (UID: "15ac71af-2277-4c22-9647-a3cca4b0a929"). InnerVolumeSpecName "kube-api-access-l9xms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:24.720620 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:24.720578    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l9xms\" (UniqueName: \"kubernetes.io/projected/15ac71af-2277-4c22-9647-a3cca4b0a929-kube-api-access-l9xms\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:25.245783 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:25.245749    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd" event={"ID":"15ac71af-2277-4c22-9647-a3cca4b0a929","Type":"ContainerDied","Data":"39861090b303acb37738672568b687a7c9e5e5f3a13c449b4358d2f756baa3a1"}
Nov 01 13:00:25.245783 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:25.245781    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"
Nov 01 13:00:25.246272 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:25.245806    4095 scope.go:122] "RemoveContainer" containerID="05213d8ce16f22d7987edc8a89cac1af0e6fc912203804c46bd97ef29d7c655d"
Nov 01 13:00:25.403116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:25.403084    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"]
Nov 01 13:00:25.407733 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:25.407704    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29892300-7djvd"]
Nov 01 13:00:26.188881 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:26.188846    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" path="/var/lib/kubelet/pods/15ac71af-2277-4c22-9647-a3cca4b0a929/volumes"
Nov 01 13:00:26.189326 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:26.189312    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" path="/var/lib/kubelet/pods/f7b40458-1b4d-46f3-ae01-d96dcf1b03aa/volumes"
Nov 01 13:00:28.543937 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.543899    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"]
Nov 01 13:00:28.544448 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544433    4095 cpu_manager.go:414] "containerMap: removing container" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerName="cleanup"
Nov 01 13:00:28.544492 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544449    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerName="cleanup"
Nov 01 13:00:28.544492 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544461    4095 cpu_manager.go:414] "containerMap: removing container" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy"
Nov 01 13:00:28.544492 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544469    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy"
Nov 01 13:00:28.544492 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544488    4095 cpu_manager.go:414] "containerMap: removing container" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerName="cleanup"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544495    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerName="cleanup"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544503    4095 cpu_manager.go:414] "containerMap: removing container" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544507    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544518    4095 cpu_manager.go:414] "containerMap: removing container" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerName="cleanup"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544522    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerName="cleanup"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544531    4095 cpu_manager.go:414] "containerMap: removing container" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerName="cleanup"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544537    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerName="cleanup"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544547    4095 cpu_manager.go:414] "containerMap: removing container" podUID="fff1b7f2-ba38-4aab-94f8-00c8f23ef154" containerName="maas-api"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544554    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fff1b7f2-ba38-4aab-94f8-00c8f23ef154" containerName="maas-api"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544562    4095 cpu_manager.go:414] "containerMap: removing container" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerName="cleanup"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544566    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerName="cleanup"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544583    4095 cpu_manager.go:414] "containerMap: removing container" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544587    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544603    4095 cpu_manager.go:414] "containerMap: removing container" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main"
Nov 01 13:00:28.544621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544608    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544684    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="c70639c6-8601-4a9c-866c-d194156300b6" containerName="main"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544696    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544707    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerName="cleanup"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544715    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5c4201-96c1-40b9-be00-774ccc27e376" containerName="main"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544722    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerName="cleanup"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544727    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544734    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerName="cleanup"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544739    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a51e92-f7f8-4b70-82ac-5439e2fc347f" containerName="istio-proxy"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544745    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerName="cleanup"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544752    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerName="cleanup"
Nov 01 13:00:28.545097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.544758    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="fff1b7f2-ba38-4aab-94f8-00c8f23ef154" containerName="maas-api"
Nov 01 13:00:28.549620 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.549602    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.599387 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.599354    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-187d06ed-openshift-default-dockercfg-h6ggd\""
Nov 01 13:00:28.650250 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.650211    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"]
Nov 01 13:00:28.698701 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.698667    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.698906 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.698736    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6735977-122d-4a0f-b5a8-b200856623e1-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.698906 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.698775    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.698906 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.698861    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.699114 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.698934    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.699114 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.699086    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.699237 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.699158    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6735977-122d-4a0f-b5a8-b200856623e1-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.699292 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.699256    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68srf\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-kube-api-access-68srf\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.699348 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.699290    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800260 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800172    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800260 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800257    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800486 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800295    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800486 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800339    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6735977-122d-4a0f-b5a8-b200856623e1-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800486 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800420    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68srf\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-kube-api-access-68srf\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800486 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800449    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800486 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800480    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800744 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800523    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6735977-122d-4a0f-b5a8-b200856623e1-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800744 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800560    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800744 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800593    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800744 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800692    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800955 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800818    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.800955 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.800930    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.801106 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.801061    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6735977-122d-4a0f-b5a8-b200856623e1-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.802486 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.802467    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.803006 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.802986    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6735977-122d-4a0f-b5a8-b200856623e1-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.856036 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.855956    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.856924 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.856896    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68srf\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-kube-api-access-68srf\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:28.863774 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:28.863747    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:29.296795 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:29.296764    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"]
Nov 01 13:00:29.297369 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:00:29.297341    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6735977_122d_4a0f_b5a8_b200856623e1.slice/crio-3ec5408aced1c2d3c4e74fbf80d145415b8fc65d9042507538dd827480acdff8 WatchSource:0}: Error finding container 3ec5408aced1c2d3c4e74fbf80d145415b8fc65d9042507538dd827480acdff8: Status 404 returned error can't find the container with id 3ec5408aced1c2d3c4e74fbf80d145415b8fc65d9042507538dd827480acdff8
Nov 01 13:00:29.299712 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:29.299682    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:00:29.299795 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:29.299743    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:00:29.299795 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:29.299770    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:00:30.282698 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:30.282661    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" event={"ID":"c6735977-122d-4a0f-b5a8-b200856623e1","Type":"ContainerStarted","Data":"e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7"}
Nov 01 13:00:30.283159 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:30.282728    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" event={"ID":"c6735977-122d-4a0f-b5a8-b200856623e1","Type":"ContainerStarted","Data":"3ec5408aced1c2d3c4e74fbf80d145415b8fc65d9042507538dd827480acdff8"}
Nov 01 13:00:30.397457 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:30.397402    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" podStartSLOduration=2.397388633 podStartE2EDuration="2.397388633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:30.395708281 +0000 UTC m=+2290.313503947" watchObservedRunningTime="2026-11-01 13:00:30.397388633 +0000 UTC m=+2290.315184298"
Nov 01 13:00:30.864203 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:30.864167    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:30.865957 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:30.865930    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 01 13:00:30.866097 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:30.865986    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 01 13:00:31.865030 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:31.864957    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 01 13:00:31.865472 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:31.865049    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 01 13:00:32.865018 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:32.864962    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 01 13:00:32.865215 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:32.865035    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 01 13:00:33.747651 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:33.747618    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"]
Nov 01 13:00:33.747854 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:33.747824    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf" podUID="94fd2915-6553-4a33-8291-4d930d27104c" containerName="maas-api" containerID="cri-o://aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9" gracePeriod=30
Nov 01 13:00:33.864876 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:33.864842    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 01 13:00:33.865048 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:33.864900    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 01 13:00:34.060692 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.060664    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 13:00:34.068836 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.068812    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fd2915-6553-4a33-8291-4d930d27104c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2915-6553-4a33-8291-4d930d27104c-openshift-service-ca\") pod \"94fd2915-6553-4a33-8291-4d930d27104c\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") "
Nov 01 13:00:34.068932 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.068891    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-maas-api-tls\") pod \"94fd2915-6553-4a33-8291-4d930d27104c\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") "
Nov 01 13:00:34.069030 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.068963    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-metrics-certs\") pod \"94fd2915-6553-4a33-8291-4d930d27104c\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") "
Nov 01 13:00:34.069073 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.069045    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fd2915-6553-4a33-8291-4d930d27104c-kube-api-access-rpj68\" (UniqueName: \"kubernetes.io/projected/94fd2915-6553-4a33-8291-4d930d27104c-kube-api-access-rpj68\") pod \"94fd2915-6553-4a33-8291-4d930d27104c\" (UID: \"94fd2915-6553-4a33-8291-4d930d27104c\") "
Nov 01 13:00:34.069246 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.069223    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fd2915-6553-4a33-8291-4d930d27104c-openshift-service-ca" pod "94fd2915-6553-4a33-8291-4d930d27104c" (UID: "94fd2915-6553-4a33-8291-4d930d27104c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:34.069496 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.069475    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2915-6553-4a33-8291-4d930d27104c-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:34.071070 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.071012    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-metrics-certs" pod "94fd2915-6553-4a33-8291-4d930d27104c" (UID: "94fd2915-6553-4a33-8291-4d930d27104c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:34.071165 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.071096    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-maas-api-tls" pod "94fd2915-6553-4a33-8291-4d930d27104c" (UID: "94fd2915-6553-4a33-8291-4d930d27104c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:00:34.071165 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.071140    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fd2915-6553-4a33-8291-4d930d27104c-kube-api-access-rpj68" pod "94fd2915-6553-4a33-8291-4d930d27104c" (UID: "94fd2915-6553-4a33-8291-4d930d27104c"). InnerVolumeSpecName "kube-api-access-rpj68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:34.171062 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.171027    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:34.171062 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.171057    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/94fd2915-6553-4a33-8291-4d930d27104c-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:34.171062 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.171068    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpj68\" (UniqueName: \"kubernetes.io/projected/94fd2915-6553-4a33-8291-4d930d27104c-kube-api-access-rpj68\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:34.345925 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.345837    4095 generic.go:356] "Generic (PLEG): container finished" podID="94fd2915-6553-4a33-8291-4d930d27104c" containerID="aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9" exitCode=0
Nov 01 13:00:34.346096 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.346047    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf" event={"ID":"94fd2915-6553-4a33-8291-4d930d27104c","Type":"ContainerDied","Data":"aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9"}
Nov 01 13:00:34.346152 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.346099    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf" event={"ID":"94fd2915-6553-4a33-8291-4d930d27104c","Type":"ContainerDied","Data":"defdc6eccd23395222088c3032cfa42bc9f2e7f7bc9aad709c4c396694f2f524"}
Nov 01 13:00:34.346152 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.346128    4095 scope.go:122] "RemoveContainer" containerID="aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9"
Nov 01 13:00:34.346304 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.346287    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"
Nov 01 13:00:34.361964 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.361933    4095 scope.go:122] "RemoveContainer" containerID="aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9"
Nov 01 13:00:34.362373 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:34.362345    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9\": container with ID starting with aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9 not found: ID does not exist" containerID="aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9"
Nov 01 13:00:34.362467 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.362386    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9"} err="failed to get container status \"aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9\": rpc error: code = NotFound desc = could not find container \"aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9\": container with ID starting with aec60722d23025353891e8618db2053292b7c44afcafcf57882a4d4f5f37d9a9 not found: ID does not exist"
Nov 01 13:00:34.529007 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.528930    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"]
Nov 01 13:00:34.532952 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.532907    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-94c94b-7c99f57bc4-6xvzf"]
Nov 01 13:00:34.864310 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.864272    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 01 13:00:34.864507 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:34.864347    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 01 13:00:35.869402 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:35.869371    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:35.869794 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:35.869756    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:35.870434 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:35.870415    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:00:36.193643 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:36.193568    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fd2915-6553-4a33-8291-4d930d27104c" path="/var/lib/kubelet/pods/94fd2915-6553-4a33-8291-4d930d27104c/volumes"
Nov 01 13:00:42.380529 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:42.380497    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"]
Nov 01 13:00:42.380944 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:42.380710    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy" containerID="cri-o://c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5" gracePeriod=30
Nov 01 13:00:43.648930 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.648893    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"]
Nov 01 13:00:43.649415 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.649402    4095 cpu_manager.go:414] "containerMap: removing container" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy"
Nov 01 13:00:43.649415 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.649415    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dc78e78-3863-44a7-b0b1-5f96566e8ec8" containerName="istio-proxy"
Nov 01 13:00:43.649487 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.649436    4095 cpu_manager.go:414] "containerMap: removing container" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerName="cleanup"
Nov 01 13:00:43.649487 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.649441    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7b40458-1b4d-46f3-ae01-d96dcf1b03aa" containerName="cleanup"
Nov 01 13:00:43.649487 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.649463    4095 cpu_manager.go:414] "containerMap: removing container" podUID="94fd2915-6553-4a33-8291-4d930d27104c" containerName="maas-api"
Nov 01 13:00:43.649487 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.649468    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fd2915-6553-4a33-8291-4d930d27104c" containerName="maas-api"
Nov 01 13:00:43.649605 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.649535    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fd2915-6553-4a33-8291-4d930d27104c" containerName="maas-api"
Nov 01 13:00:43.649605 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.649548    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="15ac71af-2277-4c22-9647-a3cca4b0a929" containerName="cleanup"
Nov 01 13:00:43.657918 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.657896    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.696490 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.696456    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"]
Nov 01 13:00:43.700702 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.700682    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.750147 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.750118    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"]
Nov 01 13:00:43.792845 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.792811    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.793015 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.792866    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.793108 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793050    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4816ead2-e1e6-4316-968a-e1e2ea46093c-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.793108 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793097    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.793278 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793258    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.793357 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793340    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.793398 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793369    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.793482 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793392    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.793482 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793422    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.793589 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793564    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.793659 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793643    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.793718 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793691    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hdvj\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-kube-api-access-8hdvj\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.793819 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793796    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.793876 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793842    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjwv5\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-kube-api-access-tjwv5\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.793934 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793876    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.793934 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793911    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.794075 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.793958    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.794075 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.794055    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.796784 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.796763    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"]
Nov 01 13:00:43.894937 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.894907    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.894937 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.894939    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hdvj\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-kube-api-access-8hdvj\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.895181 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.894989    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895181 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895016    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjwv5\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-kube-api-access-tjwv5\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895181 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895044    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.895181 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895074    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895181 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895106    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895181 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895138    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895183    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895218    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895266    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4816ead2-e1e6-4316-968a-e1e2ea46093c-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895309    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895384    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895386    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895399    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895426    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895459    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895485    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895511    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.895577 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895551    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.896211 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895641    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.896211 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895482    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-certs\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.896211 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895902    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.896211 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.895935    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-credential-socket\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.896211 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.896147    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.896491 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.896233    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-data\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.896491 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.896354    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4816ead2-e1e6-4316-968a-e1e2ea46093c-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.896491 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.896415    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istiod-ca-cert\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.897421 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.897397    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.897954 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.897934    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.898062 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.897954    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-envoy\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.898202 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.898187    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-podinfo\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.950830 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.950755    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.950999 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.950872    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjwv5\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-kube-api-access-tjwv5\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:43.951583 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.951564    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-token\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.959616 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.959594    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hdvj\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-kube-api-access-8hdvj\") pod \"e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") " pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:43.971898 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:43.971874    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:44.014524 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:44.014490    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:44.392177 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:44.392145    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"]
Nov 01 13:00:44.426461 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:00:44.426430    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccc8a7e4_94f6_4990_a8f1_7d78bcf89e2a.slice/crio-f93dfc6a8100a876c17651eb321c854ea3a829270615ef621cfb3cb8fba65622 WatchSource:0}: Error finding container f93dfc6a8100a876c17651eb321c854ea3a829270615ef621cfb3cb8fba65622: Status 404 returned error can't find the container with id f93dfc6a8100a876c17651eb321c854ea3a829270615ef621cfb3cb8fba65622
Nov 01 13:00:44.428861 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:44.428832    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:00:44.428935 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:44.428898    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:00:44.428935 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:44.428927    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:00:44.439571 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:44.439551    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"]
Nov 01 13:00:44.443103 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:44.443076    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:00:44.443166 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:44.443136    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:00:44.443220 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:44.443183    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:00:45.446993 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:45.446110    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" event={"ID":"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a","Type":"ContainerStarted","Data":"ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c"}
Nov 01 13:00:45.446993 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:45.446180    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" event={"ID":"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a","Type":"ContainerStarted","Data":"f93dfc6a8100a876c17651eb321c854ea3a829270615ef621cfb3cb8fba65622"}
Nov 01 13:00:45.455995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:45.454048    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" event={"ID":"4816ead2-e1e6-4316-968a-e1e2ea46093c","Type":"ContainerStarted","Data":"53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350"}
Nov 01 13:00:45.455995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:45.454117    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" event={"ID":"4816ead2-e1e6-4316-968a-e1e2ea46093c","Type":"ContainerStarted","Data":"0feb7fd7d991fbec24913479ac9e4e0aae687d37cc51c066024cf52bfba107b7"}
Nov 01 13:00:45.626001 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:45.624957    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" podStartSLOduration=2.624936595 podStartE2EDuration="2.624936595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:45.624385825 +0000 UTC m=+2305.542181504" watchObservedRunningTime="2026-11-01 13:00:45.624936595 +0000 UTC m=+2305.542732261"
Nov 01 13:00:45.758215 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:45.758076    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" podStartSLOduration=2.758041039 podStartE2EDuration="2.758041039s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:00:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:00:45.737480706 +0000 UTC m=+2305.655276375" watchObservedRunningTime="2026-11-01 13:00:45.758041039 +0000 UTC m=+2305.675836706"
Nov 01 13:00:45.972224 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:45.972188    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:45.973714 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:45.973687    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 13:00:45.973789 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:45.973747    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 13:00:46.024055 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:46.022540    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:46.027234 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:46.027103    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 13:00:46.027234 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:46.027165    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 13:00:46.973042 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:46.973004    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 13:00:46.973534 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:46.973056    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 13:00:47.019996 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:47.019584    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 13:00:47.019996 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:47.019648    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 13:00:47.991512 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:47.991424    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 13:00:47.992121 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:47.992066    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 13:00:48.068339 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.068307    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 13:00:48.068602 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.068560    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 13:00:48.077465 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.077441    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 13:00:48.169245 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.169208    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-podinfo\") pod \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") "
Nov 01 13:00:48.169422 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.169268    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-socket\") pod \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") "
Nov 01 13:00:48.169422 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.169351    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-certs\") pod \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") "
Nov 01 13:00:48.169494 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.169441    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istiod-ca-cert\") pod \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") "
Nov 01 13:00:48.169494 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.169465    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-token\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-token\") pod \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") "
Nov 01 13:00:48.169558 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.169505    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-kube-api-access-f6hv7\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-kube-api-access-f6hv7\") pod \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") "
Nov 01 13:00:48.169558 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.169522    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-envoy\") pod \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") "
Nov 01 13:00:48.169558 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.169543    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-data\") pod \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") "
Nov 01 13:00:48.169645 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.169571    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-credential-socket\") pod \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\" (UID: \"2d5ab7a1-0067-4280-9028-d66c40cb9cd5\") "
Nov 01 13:00:48.170385 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.170277    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istiod-ca-cert" pod "2d5ab7a1-0067-4280-9028-d66c40cb9cd5" (UID: "2d5ab7a1-0067-4280-9028-d66c40cb9cd5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:00:48.170510 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.170399    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-data" pod "2d5ab7a1-0067-4280-9028-d66c40cb9cd5" (UID: "2d5ab7a1-0067-4280-9028-d66c40cb9cd5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:48.170510 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.170454    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-credential-socket" pod "2d5ab7a1-0067-4280-9028-d66c40cb9cd5" (UID: "2d5ab7a1-0067-4280-9028-d66c40cb9cd5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:48.170936 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.170910    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-certs" pod "2d5ab7a1-0067-4280-9028-d66c40cb9cd5" (UID: "2d5ab7a1-0067-4280-9028-d66c40cb9cd5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:48.171051 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.170930    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-socket" pod "2d5ab7a1-0067-4280-9028-d66c40cb9cd5" (UID: "2d5ab7a1-0067-4280-9028-d66c40cb9cd5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:48.172783 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.172758    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-podinfo" pod "2d5ab7a1-0067-4280-9028-d66c40cb9cd5" (UID: "2d5ab7a1-0067-4280-9028-d66c40cb9cd5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:00:48.172880 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.172826    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-envoy" pod "2d5ab7a1-0067-4280-9028-d66c40cb9cd5" (UID: "2d5ab7a1-0067-4280-9028-d66c40cb9cd5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:00:48.173348 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.173318    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-token" pod "2d5ab7a1-0067-4280-9028-d66c40cb9cd5" (UID: "2d5ab7a1-0067-4280-9028-d66c40cb9cd5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:48.173571 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.173541    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-kube-api-access-f6hv7" pod "2d5ab7a1-0067-4280-9028-d66c40cb9cd5" (UID: "2d5ab7a1-0067-4280-9028-d66c40cb9cd5"). InnerVolumeSpecName "kube-api-access-f6hv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:00:48.270953 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.270884    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.271134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.271020    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.271134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.271039    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.271134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.271053    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.271134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.271066    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.271134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.271078    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.271134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.271091    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6hv7\" (UniqueName: \"kubernetes.io/projected/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-kube-api-access-f6hv7\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.271134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.271104    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.271134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.271116    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2d5ab7a1-0067-4280-9028-d66c40cb9cd5-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:00:48.475728 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.475687    4095 generic.go:356] "Generic (PLEG): container finished" podID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerID="c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5" exitCode=0
Nov 01 13:00:48.475879 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.475751    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"
Nov 01 13:00:48.475879 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.475855    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" event={"ID":"2d5ab7a1-0067-4280-9028-d66c40cb9cd5","Type":"ContainerDied","Data":"c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5"}
Nov 01 13:00:48.475945 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.475898    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff" event={"ID":"2d5ab7a1-0067-4280-9028-d66c40cb9cd5","Type":"ContainerDied","Data":"8819adc723f556c33dc8dc90e5361f268b6bf2c0ef41104470c70219beb32930"}
Nov 01 13:00:48.475945 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.475926    4095 scope.go:122] "RemoveContainer" containerID="c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5"
Nov 01 13:00:48.487784 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.487763    4095 scope.go:122] "RemoveContainer" containerID="c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5"
Nov 01 13:00:48.488069 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:48.488051    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5\": container with ID starting with c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5 not found: ID does not exist" containerID="c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5"
Nov 01 13:00:48.488155 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.488075    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5"} err="failed to get container status \"c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5\": rpc error: code = NotFound desc = could not find container \"c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5\": container with ID starting with c7ffefc6da124bab32262ca20774b9e41dba4ce330ae7a793a68dab0610643d5 not found: ID does not exist"
Nov 01 13:00:48.645993 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.645237    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"]
Nov 01 13:00:48.665993 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.657499    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-94c94b-openshift-default-666b8df787-62xff"]
Nov 01 13:00:48.978050 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.975301    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 13:00:48.978050 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:48.975373    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 13:00:49.021806 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:49.021282    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 13:00:49.021806 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:49.021338    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 13:00:49.191432 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:00:49.191398    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:00:49.976365 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:49.976317    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 13:00:49.976661 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:49.976620    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 13:00:50.016480 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:50.016445    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 13:00:50.016772 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:50.016741    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 13:00:50.253054 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:50.245157    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" path="/var/lib/kubelet/pods/2d5ab7a1-0067-4280-9028-d66c40cb9cd5/volumes"
Nov 01 13:00:50.972589 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:50.972544    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused" start-of-body=
Nov 01 13:00:50.972790 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:50.972636    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.93:15021/healthz/ready\": dial tcp 10.134.0.93:15021: connect: connection refused"
Nov 01 13:00:51.015510 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:51.015468    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body=
Nov 01 13:00:51.015661 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:51.015555    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused"
Nov 01 13:00:52.219177 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:52.219145    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:52.219718 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:52.219696    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:52.220431 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:52.220406    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:00:52.230734 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:52.230713    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:52.231172 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:52.231153    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:00:52.232315 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:00:52.232296    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:01:05.092058 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.092022    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"]
Nov 01 13:01:05.092601 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.092250    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82" podUID="6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" containerName="maas-api" containerID="cri-o://073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231" gracePeriod=30
Nov 01 13:01:05.372482 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.372455    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 13:01:05.427557 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.427530    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-metrics-certs\") pod \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") "
Nov 01 13:01:05.427713 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.427639    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-openshift-service-ca\") pod \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") "
Nov 01 13:01:05.427713 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.427693    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls\") pod \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") "
Nov 01 13:01:05.427815 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.427764    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-kube-api-access-c6x8z\" (UniqueName: \"kubernetes.io/projected/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-kube-api-access-c6x8z\") pod \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\" (UID: \"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41\") "
Nov 01 13:01:05.428028 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.428003    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-openshift-service-ca" pod "6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" (UID: "6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:01:05.428232 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.428208    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:05.429660 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.429636    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-metrics-certs" pod "6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" (UID: "6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:05.429861 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.429840    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls" pod "6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" (UID: "6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:05.429950 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.429868    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-kube-api-access-c6x8z" pod "6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" (UID: "6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41"). InnerVolumeSpecName "kube-api-access-c6x8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:05.529875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.529838    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6x8z\" (UniqueName: \"kubernetes.io/projected/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-kube-api-access-c6x8z\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:05.529875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.529874    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:05.530091 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.529887    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:05.581694 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.581641    4095 generic.go:356] "Generic (PLEG): container finished" podID="6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" containerID="073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231" exitCode=0
Nov 01 13:01:05.581875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.581702    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"
Nov 01 13:01:05.581875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.581712    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82" event={"ID":"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41","Type":"ContainerDied","Data":"073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231"}
Nov 01 13:01:05.581875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.581767    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82" event={"ID":"6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41","Type":"ContainerDied","Data":"10cc66b5d4f37c072446af776318135a2ddf0ba6cf702780311a3686c7da063f"}
Nov 01 13:01:05.581875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.581799    4095 scope.go:122] "RemoveContainer" containerID="073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231"
Nov 01 13:01:05.592222 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.592204    4095 scope.go:122] "RemoveContainer" containerID="073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231"
Nov 01 13:01:05.592465 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:01:05.592446    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231\": container with ID starting with 073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231 not found: ID does not exist" containerID="073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231"
Nov 01 13:01:05.592526 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.592478    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231"} err="failed to get container status \"073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231\": rpc error: code = NotFound desc = could not find container \"073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231\": container with ID starting with 073b8f295dc67b052f2ad27f9296d97303e9151e68756b14dcdd1e59160be231 not found: ID does not exist"
Nov 01 13:01:05.738874 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.738812    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"]
Nov 01 13:01:05.742665 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:05.742637    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-5b1bd4-97867f46d-nzh82"]
Nov 01 13:01:06.189175 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:06.189143    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" path="/var/lib/kubelet/pods/6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41/volumes"
Nov 01 13:01:08.821651 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.821617    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"]
Nov 01 13:01:08.822123 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.822109    4095 cpu_manager.go:414] "containerMap: removing container" podUID="6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" containerName="maas-api"
Nov 01 13:01:08.822175 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.822123    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" containerName="maas-api"
Nov 01 13:01:08.822175 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.822156    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy"
Nov 01 13:01:08.822175 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.822161    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy"
Nov 01 13:01:08.822273 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.822253    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d5ab7a1-0067-4280-9028-d66c40cb9cd5" containerName="istio-proxy"
Nov 01 13:01:08.822273 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.822266    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1a8c3e-3a2c-4050-b7c9-a0ea90db6e41" containerName="maas-api"
Nov 01 13:01:08.826865 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.826835    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.867678 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.867648    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.867818 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.867695    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.867818 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.867801    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.867931 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.867911    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.867993 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.867939    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.868074 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.868056    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcl9v\" (UniqueName: \"kubernetes.io/projected/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-kube-api-access-lcl9v\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.868212 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.868192    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.868266 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.868241    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.868316 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.868280    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.925052 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.925025    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"]
Nov 01 13:01:08.969224 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969189    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcl9v\" (UniqueName: \"kubernetes.io/projected/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-kube-api-access-lcl9v\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.969380 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969316    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.969380 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969353    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.969474 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969407    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.969537 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969478    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.969537 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969505    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.969628 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969534    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.969808 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969786    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.970024 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969947    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.970024 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.969805    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.970283 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.970263    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.970365 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.970350    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.970428 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.970372    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.970701 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.970677    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.971750 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.971732    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:08.971965 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:08.971950    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:09.025981 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:09.025939    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:09.026184 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:09.026162    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcl9v\" (UniqueName: \"kubernetes.io/projected/7352735c-f1a9-4480-bca7-c9cbe3f9e8e2-kube-api-access-lcl9v\") pod \"maas-default-gateway-openshift-default-7df854bf67-29nw6\" (UID: \"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:09.139799 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:09.139765    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:09.563889 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:09.563860    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"]
Nov 01 13:01:09.564553 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:01:09.564527    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7352735c_f1a9_4480_bca7_c9cbe3f9e8e2.slice/crio-16c36087d6c121c73d1b7e6edc28d26f77df4d9716c410d704ec5e8bf6ef30eb WatchSource:0}: Error finding container 16c36087d6c121c73d1b7e6edc28d26f77df4d9716c410d704ec5e8bf6ef30eb: Status 404 returned error can't find the container with id 16c36087d6c121c73d1b7e6edc28d26f77df4d9716c410d704ec5e8bf6ef30eb
Nov 01 13:01:09.566943 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:09.566911    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:01:09.567036 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:09.567004    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:01:09.567080 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:09.567038    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:01:09.614010 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:09.609197    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" event={"ID":"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2","Type":"ContainerStarted","Data":"16c36087d6c121c73d1b7e6edc28d26f77df4d9716c410d704ec5e8bf6ef30eb"}
Nov 01 13:01:10.633657 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:10.633613    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" event={"ID":"7352735c-f1a9-4480-bca7-c9cbe3f9e8e2","Type":"ContainerStarted","Data":"d8340b741aa048f2d97f42a310aace8767d0b041a3dc18139f028da311c95e19"}
Nov 01 13:01:10.796439 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:10.796385    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" podStartSLOduration=2.796370737 podStartE2EDuration="2.796370737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:01:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:01:10.793696114 +0000 UTC m=+2330.711491780" watchObservedRunningTime="2026-11-01 13:01:10.796370737 +0000 UTC m=+2330.714166400"
Nov 01 13:01:11.140042 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:11.140007    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:11.141806 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:11.141775    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-29nw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 13:01:11.141961 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:11.141832    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" podUID="7352735c-f1a9-4480-bca7-c9cbe3f9e8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 13:01:12.140079 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:12.140041    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-29nw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 13:01:12.140480 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:12.140101    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" podUID="7352735c-f1a9-4480-bca7-c9cbe3f9e8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 13:01:13.140710 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:13.140676    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-29nw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 13:01:13.141120 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:13.140730    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" podUID="7352735c-f1a9-4480-bca7-c9cbe3f9e8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 13:01:14.140705 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:14.140664    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-29nw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 13:01:14.140869 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:14.140720    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" podUID="7352735c-f1a9-4480-bca7-c9cbe3f9e8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 13:01:14.933719 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:14.933685    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"]
Nov 01 13:01:14.934008 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:14.933934    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy" containerID="cri-o://aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962" gracePeriod=30
Nov 01 13:01:15.140242 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:15.140201    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-29nw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 13:01:15.140416 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:15.140258    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" podUID="7352735c-f1a9-4480-bca7-c9cbe3f9e8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 13:01:16.140158 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:16.140123    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-29nw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 13:01:16.140660 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:16.140200    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" podUID="7352735c-f1a9-4480-bca7-c9cbe3f9e8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 13:01:17.141011 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:17.140956    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-29nw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused" start-of-body=
Nov 01 13:01:17.141511 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:17.141053    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6" podUID="7352735c-f1a9-4480-bca7-c9cbe3f9e8e2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.95:15021/healthz/ready\": dial tcp 10.134.0.95:15021: connect: connection refused"
Nov 01 13:01:18.175579 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:18.175549    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:18.176094 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:18.176075    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:18.176912 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:18.176890    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-29nw6"
Nov 01 13:01:18.838507 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:18.838469    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"]
Nov 01 13:01:18.838699 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:18.838672    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4" podUID="b4328f10-aaee-41f8-8c0c-5fb584f00662" containerName="discovery" containerID="cri-o://f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8" gracePeriod=30
Nov 01 13:01:19.102495 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.102424    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 13:01:19.118237 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:01:19.118202    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:01:19.222782 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.222743    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-dns-cert\") pod \"b4328f10-aaee-41f8-8c0c-5fb584f00662\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") "
Nov 01 13:01:19.223312 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.222811    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-token\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-token\") pod \"b4328f10-aaee-41f8-8c0c-5fb584f00662\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") "
Nov 01 13:01:19.223312 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.222887    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-kube-api-access-pmx2z\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-kube-api-access-pmx2z\") pod \"b4328f10-aaee-41f8-8c0c-5fb584f00662\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") "
Nov 01 13:01:19.223312 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.222945    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4328f10-aaee-41f8-8c0c-5fb584f00662-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4328f10-aaee-41f8-8c0c-5fb584f00662-local-certs\") pod \"b4328f10-aaee-41f8-8c0c-5fb584f00662\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") "
Nov 01 13:01:19.223312 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.223063    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-ca-configmap\") pod \"b4328f10-aaee-41f8-8c0c-5fb584f00662\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") "
Nov 01 13:01:19.223312 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.223087    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-cacerts\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-cacerts\") pod \"b4328f10-aaee-41f8-8c0c-5fb584f00662\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") "
Nov 01 13:01:19.223312 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.223171    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-kubeconfig\") pod \"b4328f10-aaee-41f8-8c0c-5fb584f00662\" (UID: \"b4328f10-aaee-41f8-8c0c-5fb584f00662\") "
Nov 01 13:01:19.224139 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.224047    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-ca-configmap" pod "b4328f10-aaee-41f8-8c0c-5fb584f00662" (UID: "b4328f10-aaee-41f8-8c0c-5fb584f00662"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:01:19.225606 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.225572    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-kube-api-access-pmx2z" pod "b4328f10-aaee-41f8-8c0c-5fb584f00662" (UID: "b4328f10-aaee-41f8-8c0c-5fb584f00662"). InnerVolumeSpecName "kube-api-access-pmx2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:19.225832 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.225800    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-cacerts" pod "b4328f10-aaee-41f8-8c0c-5fb584f00662" (UID: "b4328f10-aaee-41f8-8c0c-5fb584f00662"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:19.225832 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.225816    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-dns-cert" pod "b4328f10-aaee-41f8-8c0c-5fb584f00662" (UID: "b4328f10-aaee-41f8-8c0c-5fb584f00662"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:19.226142 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.226123    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-kubeconfig" pod "b4328f10-aaee-41f8-8c0c-5fb584f00662" (UID: "b4328f10-aaee-41f8-8c0c-5fb584f00662"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:19.226337 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.226322    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-token" pod "b4328f10-aaee-41f8-8c0c-5fb584f00662" (UID: "b4328f10-aaee-41f8-8c0c-5fb584f00662"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:19.226520 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.226503    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4328f10-aaee-41f8-8c0c-5fb584f00662-local-certs" pod "b4328f10-aaee-41f8-8c0c-5fb584f00662" (UID: "b4328f10-aaee-41f8-8c0c-5fb584f00662"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:19.325421 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.325388    4095 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-kubeconfig\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:19.325421 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.325423    4095 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-dns-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:19.325634 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.325440    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:19.325634 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.325454    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmx2z\" (UniqueName: \"kubernetes.io/projected/b4328f10-aaee-41f8-8c0c-5fb584f00662-kube-api-access-pmx2z\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:19.325634 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.325465    4095 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4328f10-aaee-41f8-8c0c-5fb584f00662-local-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:19.325634 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.325474    4095 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/b4328f10-aaee-41f8-8c0c-5fb584f00662-istio-csr-ca-configmap\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:19.325634 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.325483    4095 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/b4328f10-aaee-41f8-8c0c-5fb584f00662-cacerts\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:19.693873 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.693838    4095 generic.go:356] "Generic (PLEG): container finished" podID="b4328f10-aaee-41f8-8c0c-5fb584f00662" containerID="f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8" exitCode=0
Nov 01 13:01:19.694114 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.693885    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"
Nov 01 13:01:19.694114 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.693927    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4" event={"ID":"b4328f10-aaee-41f8-8c0c-5fb584f00662","Type":"ContainerDied","Data":"f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8"}
Nov 01 13:01:19.694114 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.693988    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4" event={"ID":"b4328f10-aaee-41f8-8c0c-5fb584f00662","Type":"ContainerDied","Data":"24fa87c5696143247b212280a9fe0479b8b602896ad3d59e9516e7d709973a4b"}
Nov 01 13:01:19.694114 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.694010    4095 scope.go:122] "RemoveContainer" containerID="f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8"
Nov 01 13:01:19.706542 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.706464    4095 scope.go:122] "RemoveContainer" containerID="f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8"
Nov 01 13:01:19.706871 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:01:19.706807    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8\": container with ID starting with f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8 not found: ID does not exist" containerID="f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8"
Nov 01 13:01:19.706871 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.706840    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8"} err="failed to get container status \"f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8\": rpc error: code = NotFound desc = could not find container \"f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8\": container with ID starting with f5435765ffe75115a84ada1aba0b5ad2c0b880ceac3263d317a59d62ae62bfd8 not found: ID does not exist"
Nov 01 13:01:19.851393 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.851348    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"]
Nov 01 13:01:19.898424 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:19.898392    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2n9j4"]
Nov 01 13:01:20.190510 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.190484    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4328f10-aaee-41f8-8c0c-5fb584f00662" path="/var/lib/kubelet/pods/b4328f10-aaee-41f8-8c0c-5fb584f00662/volumes"
Nov 01 13:01:20.200362 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.200309    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 13:01:20.338379 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338344    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-envoy\") pod \"af74efd4-549a-4129-9c9e-01e27494f9e4\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") "
Nov 01 13:01:20.339148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338396    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-token\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-token\") pod \"af74efd4-549a-4129-9c9e-01e27494f9e4\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") "
Nov 01 13:01:20.339148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338441    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-kube-api-access-h8r94\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-kube-api-access-h8r94\") pod \"af74efd4-549a-4129-9c9e-01e27494f9e4\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") "
Nov 01 13:01:20.339148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338474    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-data\") pod \"af74efd4-549a-4129-9c9e-01e27494f9e4\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") "
Nov 01 13:01:20.339148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338522    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af74efd4-549a-4129-9c9e-01e27494f9e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af74efd4-549a-4129-9c9e-01e27494f9e4-istiod-ca-cert\") pod \"af74efd4-549a-4129-9c9e-01e27494f9e4\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") "
Nov 01 13:01:20.339148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338584    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-podinfo\") pod \"af74efd4-549a-4129-9c9e-01e27494f9e4\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") "
Nov 01 13:01:20.339148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338611    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-socket\") pod \"af74efd4-549a-4129-9c9e-01e27494f9e4\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") "
Nov 01 13:01:20.339148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338637    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-certs\") pod \"af74efd4-549a-4129-9c9e-01e27494f9e4\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") "
Nov 01 13:01:20.339148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338687    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-credential-socket\") pod \"af74efd4-549a-4129-9c9e-01e27494f9e4\" (UID: \"af74efd4-549a-4129-9c9e-01e27494f9e4\") "
Nov 01 13:01:20.339148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.338937    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-data" pod "af74efd4-549a-4129-9c9e-01e27494f9e4" (UID: "af74efd4-549a-4129-9c9e-01e27494f9e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:20.339934 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.339909    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af74efd4-549a-4129-9c9e-01e27494f9e4-istiod-ca-cert" pod "af74efd4-549a-4129-9c9e-01e27494f9e4" (UID: "af74efd4-549a-4129-9c9e-01e27494f9e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:01:20.340682 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.340655    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-envoy" pod "af74efd4-549a-4129-9c9e-01e27494f9e4" (UID: "af74efd4-549a-4129-9c9e-01e27494f9e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:20.340827 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.340805    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:20.340891 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.340824    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:20.340891 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.340842    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af74efd4-549a-4129-9c9e-01e27494f9e4-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:20.341618 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.341592    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-token" pod "af74efd4-549a-4129-9c9e-01e27494f9e4" (UID: "af74efd4-549a-4129-9c9e-01e27494f9e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:20.341909 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.341888    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-certs" pod "af74efd4-549a-4129-9c9e-01e27494f9e4" (UID: "af74efd4-549a-4129-9c9e-01e27494f9e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:20.342103 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.342079    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-podinfo" pod "af74efd4-549a-4129-9c9e-01e27494f9e4" (UID: "af74efd4-549a-4129-9c9e-01e27494f9e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:01:20.342103 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.342093    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-socket" pod "af74efd4-549a-4129-9c9e-01e27494f9e4" (UID: "af74efd4-549a-4129-9c9e-01e27494f9e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:20.342317 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.342297    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-credential-socket" pod "af74efd4-549a-4129-9c9e-01e27494f9e4" (UID: "af74efd4-549a-4129-9c9e-01e27494f9e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:01:20.343248 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.343231    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-kube-api-access-h8r94" pod "af74efd4-549a-4129-9c9e-01e27494f9e4" (UID: "af74efd4-549a-4129-9c9e-01e27494f9e4"). InnerVolumeSpecName "kube-api-access-h8r94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:20.441790 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.441760    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:20.441790 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.441786    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8r94\" (UniqueName: \"kubernetes.io/projected/af74efd4-549a-4129-9c9e-01e27494f9e4-kube-api-access-h8r94\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:20.441790 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.441798    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af74efd4-549a-4129-9c9e-01e27494f9e4-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:20.442041 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.441807    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:20.442041 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.441816    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:20.442041 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.441824    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af74efd4-549a-4129-9c9e-01e27494f9e4-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:20.700361 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.700327    4095 generic.go:356] "Generic (PLEG): container finished" podID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerID="aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962" exitCode=0
Nov 01 13:01:20.700559 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.700377    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"
Nov 01 13:01:20.700559 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.700411    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" event={"ID":"af74efd4-549a-4129-9c9e-01e27494f9e4","Type":"ContainerDied","Data":"aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962"}
Nov 01 13:01:20.700559 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.700462    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx" event={"ID":"af74efd4-549a-4129-9c9e-01e27494f9e4","Type":"ContainerDied","Data":"649a30545f33cd412f0f79e692ef6bfad8823fb2583265f7a3bc5ed6f192c754"}
Nov 01 13:01:20.700559 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.700495    4095 scope.go:122] "RemoveContainer" containerID="aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962"
Nov 01 13:01:20.710736 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.710719    4095 scope.go:122] "RemoveContainer" containerID="aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962"
Nov 01 13:01:20.711049 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:01:20.711030    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962\": container with ID starting with aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962 not found: ID does not exist" containerID="aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962"
Nov 01 13:01:20.711101 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.711057    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962"} err="failed to get container status \"aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962\": rpc error: code = NotFound desc = could not find container \"aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962\": container with ID starting with aa1a8aac9b33fe1380c2b9abb5621ed346712af200b55ac0cb74a7ee95b81962 not found: ID does not exist"
Nov 01 13:01:20.856750 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.856709    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"]
Nov 01 13:01:20.904463 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:20.904436    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-5b1bd4-openshift-default-5c86df9c59-mlzpx"]
Nov 01 13:01:22.188763 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:22.188729    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" path="/var/lib/kubelet/pods/af74efd4-549a-4129-9c9e-01e27494f9e4/volumes"
Nov 01 13:01:23.945485 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:23.945446    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"]
Nov 01 13:01:23.946307 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:23.946285    4095 cpu_manager.go:414] "containerMap: removing container" podUID="b4328f10-aaee-41f8-8c0c-5fb584f00662" containerName="discovery"
Nov 01 13:01:23.946367 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:23.946312    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4328f10-aaee-41f8-8c0c-5fb584f00662" containerName="discovery"
Nov 01 13:01:23.946367 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:23.946359    4095 cpu_manager.go:414] "containerMap: removing container" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy"
Nov 01 13:01:23.946438 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:23.946368    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy"
Nov 01 13:01:23.946524 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:23.946509    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4328f10-aaee-41f8-8c0c-5fb584f00662" containerName="discovery"
Nov 01 13:01:23.946557 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:23.946536    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="af74efd4-549a-4129-9c9e-01e27494f9e4" containerName="istio-proxy"
Nov 01 13:01:23.952588 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:23.952569    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.053942 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.053906    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"]
Nov 01 13:01:24.085449 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.085410    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.085605 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.085480    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.085605 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.085560    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.085682 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.085633    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.085722 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.085693    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.085757 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.085733    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.085788 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.085755    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw4hp\" (UniqueName: \"kubernetes.io/projected/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-kube-api-access-lw4hp\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.085897 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.085873    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.085945 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.085929    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.186988 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.186943    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187192 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.186999    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187192 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187055    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187315 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187190    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187315 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187272    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187433 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187313    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw4hp\" (UniqueName: \"kubernetes.io/projected/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-kube-api-access-lw4hp\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187433 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187358    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187433 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187401    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187433 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187409    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187702 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187479    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187702 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187502    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.187824 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.187745    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.188091 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.188067    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.188206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.188135    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.189608 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.189578    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.189915 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.189895    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.241864 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.241792    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.242120 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.242098    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw4hp\" (UniqueName: \"kubernetes.io/projected/7adcf7b8-dcff-4e54-9918-98dec1cf66b4-kube-api-access-lw4hp\") pod \"maas-default-gateway-openshift-default-7df854bf67-b4vmw\" (UID: \"7adcf7b8-dcff-4e54-9918-98dec1cf66b4\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.273111 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.273083    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:24.688316 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.688286    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"]
Nov 01 13:01:24.689155 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:01:24.689126    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7adcf7b8_dcff_4e54_9918_98dec1cf66b4.slice/crio-f322d8ea7521a54915cd2d56b3b88a02a1d4d6de5114ae243de0f207ccad0cb8 WatchSource:0}: Error finding container f322d8ea7521a54915cd2d56b3b88a02a1d4d6de5114ae243de0f207ccad0cb8: Status 404 returned error can't find the container with id f322d8ea7521a54915cd2d56b3b88a02a1d4d6de5114ae243de0f207ccad0cb8
Nov 01 13:01:24.691629 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.691597    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:01:24.691705 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.691679    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:01:24.691743 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.691708    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:01:24.723294 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:24.723267    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" event={"ID":"7adcf7b8-dcff-4e54-9918-98dec1cf66b4","Type":"ContainerStarted","Data":"f322d8ea7521a54915cd2d56b3b88a02a1d4d6de5114ae243de0f207ccad0cb8"}
Nov 01 13:01:25.733432 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:25.733390    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" event={"ID":"7adcf7b8-dcff-4e54-9918-98dec1cf66b4","Type":"ContainerStarted","Data":"5e4af13a855b733e8216cd8995dad6074925d5d792d59093c0a7cb1e7fcb8bd5"}
Nov 01 13:01:25.946084 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:25.946031    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" podStartSLOduration=2.94601434 podStartE2EDuration="2.94601434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:01:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:01:25.944330029 +0000 UTC m=+2345.862125696" watchObservedRunningTime="2026-11-01 13:01:25.94601434 +0000 UTC m=+2345.863810005"
Nov 01 13:01:26.273438 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:26.273408    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:26.274698 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:26.274669    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b4vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 01 13:01:26.274836 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:26.274716    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" podUID="7adcf7b8-dcff-4e54-9918-98dec1cf66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 01 13:01:27.274460 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:27.274425    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b4vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 01 13:01:27.275033 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:27.274488    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" podUID="7adcf7b8-dcff-4e54-9918-98dec1cf66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 01 13:01:28.273455 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:28.273419    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b4vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 01 13:01:28.273662 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:28.273477    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" podUID="7adcf7b8-dcff-4e54-9918-98dec1cf66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 01 13:01:29.274357 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.274320    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b4vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 01 13:01:29.274862 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.274375    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" podUID="7adcf7b8-dcff-4e54-9918-98dec1cf66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 01 13:01:29.537570 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.537488    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"]
Nov 01 13:01:29.542258 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.542239    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.591750 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.591721    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d9af8c-s2qzb\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.592082 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.592061    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrbk\" (UniqueName: \"kubernetes.io/projected/2fdbf1d4-410e-4731-a492-3c425e5090e9-kube-api-access-grrbk\") pod \"maas-api-revoke-sub-41d9af8c-s2qzb\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.592194 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.592107    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d9af8c-s2qzb\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.593864 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.593843    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"]
Nov 01 13:01:29.643993 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.638337    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:01:29.643993 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.638448    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:01:29.693058 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.693029    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrbk\" (UniqueName: \"kubernetes.io/projected/2fdbf1d4-410e-4731-a492-3c425e5090e9-kube-api-access-grrbk\") pod \"maas-api-revoke-sub-41d9af8c-s2qzb\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.693257 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.693072    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d9af8c-s2qzb\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.693257 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.693150    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d9af8c-s2qzb\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.693745 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.693722    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41d9af8c-s2qzb\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.695437 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.695414    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41d9af8c-s2qzb\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.747589 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.747558    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrbk\" (UniqueName: \"kubernetes.io/projected/2fdbf1d4-410e-4731-a492-3c425e5090e9-kube-api-access-grrbk\") pod \"maas-api-revoke-sub-41d9af8c-s2qzb\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:29.858436 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:29.858407    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:30.250918 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:30.250888    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"]
Nov 01 13:01:30.260288 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:01:30.258429    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fdbf1d4_410e_4731_a492_3c425e5090e9.slice/crio-b4262301a31abbcf36a6d96dc74f26bb5114f675e55b1e24aafda59df139c656 WatchSource:0}: Error finding container b4262301a31abbcf36a6d96dc74f26bb5114f675e55b1e24aafda59df139c656: Status 404 returned error can't find the container with id b4262301a31abbcf36a6d96dc74f26bb5114f675e55b1e24aafda59df139c656
Nov 01 13:01:30.275389 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:30.275354    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b4vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 01 13:01:30.275813 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:30.275403    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" podUID="7adcf7b8-dcff-4e54-9918-98dec1cf66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 01 13:01:30.763657 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:30.763618    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb" event={"ID":"2fdbf1d4-410e-4731-a492-3c425e5090e9","Type":"ContainerStarted","Data":"cfda49805ba3f70cacf8d2d6d4c934f6463d971b7501be56efc8b53668eac1d1"}
Nov 01 13:01:30.763832 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:30.763675    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb" event={"ID":"2fdbf1d4-410e-4731-a492-3c425e5090e9","Type":"ContainerStarted","Data":"b4262301a31abbcf36a6d96dc74f26bb5114f675e55b1e24aafda59df139c656"}
Nov 01 13:01:30.872084 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:30.872031    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb" podStartSLOduration=1.8720167509999999 podStartE2EDuration="1.872016751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:01:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:01:30.871808731 +0000 UTC m=+2350.789604397" watchObservedRunningTime="2026-11-01 13:01:30.872016751 +0000 UTC m=+2350.789812417"
Nov 01 13:01:31.274448 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:31.274370    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b4vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 01 13:01:31.274448 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:31.274423    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" podUID="7adcf7b8-dcff-4e54-9918-98dec1cf66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 01 13:01:31.770868 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:31.770832    4095 generic.go:356] "Generic (PLEG): container finished" podID="2fdbf1d4-410e-4731-a492-3c425e5090e9" containerID="cfda49805ba3f70cacf8d2d6d4c934f6463d971b7501be56efc8b53668eac1d1" exitCode=0
Nov 01 13:01:31.771279 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:31.770918    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb" event={"ID":"2fdbf1d4-410e-4731-a492-3c425e5090e9","Type":"ContainerDied","Data":"cfda49805ba3f70cacf8d2d6d4c934f6463d971b7501be56efc8b53668eac1d1"}
Nov 01 13:01:32.273876 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:32.273836    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-b4vmw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body=
Nov 01 13:01:32.274095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:32.273899    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw" podUID="7adcf7b8-dcff-4e54-9918-98dec1cf66b4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused"
Nov 01 13:01:32.945724 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:32.945701    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:33.059797 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.059710    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-xks-ca\") pod \"2fdbf1d4-410e-4731-a492-3c425e5090e9\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") "
Nov 01 13:01:33.060036 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.059906    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-openshift-ca\") pod \"2fdbf1d4-410e-4731-a492-3c425e5090e9\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") "
Nov 01 13:01:33.060036 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.060004    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fdbf1d4-410e-4731-a492-3c425e5090e9-kube-api-access-grrbk\" (UniqueName: \"kubernetes.io/projected/2fdbf1d4-410e-4731-a492-3c425e5090e9-kube-api-access-grrbk\") pod \"2fdbf1d4-410e-4731-a492-3c425e5090e9\" (UID: \"2fdbf1d4-410e-4731-a492-3c425e5090e9\") "
Nov 01 13:01:33.062831 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.061497    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-openshift-ca" pod "2fdbf1d4-410e-4731-a492-3c425e5090e9" (UID: "2fdbf1d4-410e-4731-a492-3c425e5090e9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:01:33.074992 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.064530    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-xks-ca" pod "2fdbf1d4-410e-4731-a492-3c425e5090e9" (UID: "2fdbf1d4-410e-4731-a492-3c425e5090e9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:01:33.074992 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.071960    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fdbf1d4-410e-4731-a492-3c425e5090e9-kube-api-access-grrbk" pod "2fdbf1d4-410e-4731-a492-3c425e5090e9" (UID: "2fdbf1d4-410e-4731-a492-3c425e5090e9"). InnerVolumeSpecName "kube-api-access-grrbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:01:33.161733 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.161689    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-xks-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.161733 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.161735    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fdbf1d4-410e-4731-a492-3c425e5090e9-maas-api-openshift-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.162104 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.161754    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grrbk\" (UniqueName: \"kubernetes.io/projected/2fdbf1d4-410e-4731-a492-3c425e5090e9-kube-api-access-grrbk\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:01:33.322419 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.322299    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:33.322732 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.322701    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:33.323731 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.323708    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-b4vmw"
Nov 01 13:01:33.783085 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.783050    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb" event={"ID":"2fdbf1d4-410e-4731-a492-3c425e5090e9","Type":"ContainerDied","Data":"b4262301a31abbcf36a6d96dc74f26bb5114f675e55b1e24aafda59df139c656"}
Nov 01 13:01:33.783085 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.783074    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"
Nov 01 13:01:33.783291 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:01:33.783104    4095 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4262301a31abbcf36a6d96dc74f26bb5114f675e55b1e24aafda59df139c656"
Nov 01 13:01:49.113418 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:01:49.113371    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:02:03.827587 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:03.827507    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"]
Nov 01 13:02:03.828050 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:03.827739    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm" podUID="18010a26-70d2-46fb-a44e-3bedc5c1d6b1" containerName="maas-api" containerID="cri-o://30775287d46fe41030475017bb0cbf2126197d3bd3ca31973110850c2d4829a7" gracePeriod=30
Nov 01 13:02:03.947102 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:03.947076    4095 generic.go:356] "Generic (PLEG): container finished" podID="18010a26-70d2-46fb-a44e-3bedc5c1d6b1" containerID="30775287d46fe41030475017bb0cbf2126197d3bd3ca31973110850c2d4829a7" exitCode=0
Nov 01 13:02:03.947199 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:03.947145    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm" event={"ID":"18010a26-70d2-46fb-a44e-3bedc5c1d6b1","Type":"ContainerDied","Data":"30775287d46fe41030475017bb0cbf2126197d3bd3ca31973110850c2d4829a7"}
Nov 01 13:02:03.985938 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:03.985902    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm" podUID="18010a26-70d2-46fb-a44e-3bedc5c1d6b1" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.48:8443/health\": dial tcp 10.134.0.48:8443: connect: connection refused"
Nov 01 13:02:04.102692 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.102662    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 13:02:04.241303 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.241271    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-openshift-service-ca\") pod \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") "
Nov 01 13:02:04.241505 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.241327    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-metrics-certs\") pod \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") "
Nov 01 13:02:04.241505 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.241372    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls\") pod \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") "
Nov 01 13:02:04.241505 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.241440    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-kube-api-access-mh7cb\" (UniqueName: \"kubernetes.io/projected/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-kube-api-access-mh7cb\") pod \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\" (UID: \"18010a26-70d2-46fb-a44e-3bedc5c1d6b1\") "
Nov 01 13:02:04.241758 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.241729    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-openshift-service-ca" pod "18010a26-70d2-46fb-a44e-3bedc5c1d6b1" (UID: "18010a26-70d2-46fb-a44e-3bedc5c1d6b1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:04.242118 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.242096    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:04.244054 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.244029    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-metrics-certs" pod "18010a26-70d2-46fb-a44e-3bedc5c1d6b1" (UID: "18010a26-70d2-46fb-a44e-3bedc5c1d6b1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:02:04.244163 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.244060    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls" pod "18010a26-70d2-46fb-a44e-3bedc5c1d6b1" (UID: "18010a26-70d2-46fb-a44e-3bedc5c1d6b1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:02:04.244233 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.244197    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-kube-api-access-mh7cb" pod "18010a26-70d2-46fb-a44e-3bedc5c1d6b1" (UID: "18010a26-70d2-46fb-a44e-3bedc5c1d6b1"). InnerVolumeSpecName "kube-api-access-mh7cb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:04.343528 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.343449    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mh7cb\" (UniqueName: \"kubernetes.io/projected/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-kube-api-access-mh7cb\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:04.343528 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.343478    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:04.343528 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.343492    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/18010a26-70d2-46fb-a44e-3bedc5c1d6b1-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:04.953805 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.953771    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"
Nov 01 13:02:04.953805 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.953796    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm" event={"ID":"18010a26-70d2-46fb-a44e-3bedc5c1d6b1","Type":"ContainerDied","Data":"4e5f3f2d0c1bbf455a7f481ac0fbac3e1b1cf9a5dfc1901eec0078a40ad63dd3"}
Nov 01 13:02:04.954433 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:04.953860    4095 scope.go:122] "RemoveContainer" containerID="30775287d46fe41030475017bb0cbf2126197d3bd3ca31973110850c2d4829a7"
Nov 01 13:02:05.113021 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:05.112983    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"]
Nov 01 13:02:05.160049 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:05.160017    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-9514b0-644f6bb497-dv9pm"]
Nov 01 13:02:06.189398 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:06.189367    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18010a26-70d2-46fb-a44e-3bedc5c1d6b1" path="/var/lib/kubelet/pods/18010a26-70d2-46fb-a44e-3bedc5c1d6b1/volumes"
Nov 01 13:02:19.115920 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:02:19.115878    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:02:20.520851 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:20.520818    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:02:20.529474 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:20.529450    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:02:24.450995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:24.449083    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"]
Nov 01 13:02:24.450995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:24.449711    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy" containerID="cri-o://df25cfc45df80befabf5d5a7a9d9f3beb798783553b48f244883453f7d6b21f8" gracePeriod=30
Nov 01 13:02:24.919995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:24.918091    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gjsds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 13:02:24.919995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:24.918178    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" podUID="490cb0f8-5280-4505-9141-43cd22eb0cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:02:30.101706 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.101666    4095 generic.go:356] "Generic (PLEG): container finished" podID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerID="df25cfc45df80befabf5d5a7a9d9f3beb798783553b48f244883453f7d6b21f8" exitCode=0
Nov 01 13:02:30.102141 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.101721    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" event={"ID":"94d15666-ddd7-4270-8e31-4630a4b89b39","Type":"ContainerDied","Data":"df25cfc45df80befabf5d5a7a9d9f3beb798783553b48f244883453f7d6b21f8"}
Nov 01 13:02:30.102141 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.101768    4095 scope.go:122] "RemoveContainer" containerID="22236c497292401b9ca72a58439df4ddc2238762a796794ded066bc01690895c"
Nov 01 13:02:30.133410 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.133385    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 13:02:30.320492 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.320403    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-podinfo\") pod \"94d15666-ddd7-4270-8e31-4630a4b89b39\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") "
Nov 01 13:02:30.320492 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.320454    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94d15666-ddd7-4270-8e31-4630a4b89b39-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d15666-ddd7-4270-8e31-4630a4b89b39-istiod-ca-cert\") pod \"94d15666-ddd7-4270-8e31-4630a4b89b39\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") "
Nov 01 13:02:30.320704 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.320559    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-envoy\") pod \"94d15666-ddd7-4270-8e31-4630a4b89b39\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") "
Nov 01 13:02:30.320704 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.320602    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-token\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-token\") pod \"94d15666-ddd7-4270-8e31-4630a4b89b39\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") "
Nov 01 13:02:30.320704 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.320620    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-socket\") pod \"94d15666-ddd7-4270-8e31-4630a4b89b39\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") "
Nov 01 13:02:30.320704 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.320667    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-certs\") pod \"94d15666-ddd7-4270-8e31-4630a4b89b39\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") "
Nov 01 13:02:30.320704 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.320698    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-credential-socket\") pod \"94d15666-ddd7-4270-8e31-4630a4b89b39\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") "
Nov 01 13:02:30.321008 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.320776    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-kube-api-access-2tx46\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-kube-api-access-2tx46\") pod \"94d15666-ddd7-4270-8e31-4630a4b89b39\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") "
Nov 01 13:02:30.321008 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.320818    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-data\") pod \"94d15666-ddd7-4270-8e31-4630a4b89b39\" (UID: \"94d15666-ddd7-4270-8e31-4630a4b89b39\") "
Nov 01 13:02:30.321144 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321008    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-socket" pod "94d15666-ddd7-4270-8e31-4630a4b89b39" (UID: "94d15666-ddd7-4270-8e31-4630a4b89b39"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:30.321144 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321016    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94d15666-ddd7-4270-8e31-4630a4b89b39-istiod-ca-cert" pod "94d15666-ddd7-4270-8e31-4630a4b89b39" (UID: "94d15666-ddd7-4270-8e31-4630a4b89b39"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:30.321144 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321019    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-certs" pod "94d15666-ddd7-4270-8e31-4630a4b89b39" (UID: "94d15666-ddd7-4270-8e31-4630a4b89b39"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:30.321320 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321209    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-credential-socket" pod "94d15666-ddd7-4270-8e31-4630a4b89b39" (UID: "94d15666-ddd7-4270-8e31-4630a4b89b39"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:30.321320 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321260    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-data" pod "94d15666-ddd7-4270-8e31-4630a4b89b39" (UID: "94d15666-ddd7-4270-8e31-4630a4b89b39"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:30.321460 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321440    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:30.321527 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321468    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:30.321527 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321482    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:30.321527 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321496    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94d15666-ddd7-4270-8e31-4630a4b89b39-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:30.321527 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.321510    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:30.322802 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.322779    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-podinfo" pod "94d15666-ddd7-4270-8e31-4630a4b89b39" (UID: "94d15666-ddd7-4270-8e31-4630a4b89b39"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:02:30.322921 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.322868    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-envoy" pod "94d15666-ddd7-4270-8e31-4630a4b89b39" (UID: "94d15666-ddd7-4270-8e31-4630a4b89b39"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:30.322921 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.322899    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-token" pod "94d15666-ddd7-4270-8e31-4630a4b89b39" (UID: "94d15666-ddd7-4270-8e31-4630a4b89b39"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:30.323135 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.323112    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-kube-api-access-2tx46" pod "94d15666-ddd7-4270-8e31-4630a4b89b39" (UID: "94d15666-ddd7-4270-8e31-4630a4b89b39"). InnerVolumeSpecName "kube-api-access-2tx46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:30.422412 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.422380    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:30.422412 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.422407    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:30.422615 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.422423    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tx46\" (UniqueName: \"kubernetes.io/projected/94d15666-ddd7-4270-8e31-4630a4b89b39-kube-api-access-2tx46\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:30.422615 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:30.422433    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94d15666-ddd7-4270-8e31-4630a4b89b39-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:31.112175 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:31.112144    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"
Nov 01 13:02:31.112685 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:31.112185    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp" event={"ID":"94d15666-ddd7-4270-8e31-4630a4b89b39","Type":"ContainerDied","Data":"296120bbaad9c875ad497d05fab15555e8efa1c3f32060b5f1465e19d6b7af7e"}
Nov 01 13:02:31.112685 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:31.112252    4095 scope.go:122] "RemoveContainer" containerID="df25cfc45df80befabf5d5a7a9d9f3beb798783553b48f244883453f7d6b21f8"
Nov 01 13:02:31.317450 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:31.317419    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"]
Nov 01 13:02:31.365601 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:31.365527    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-06fd6b-openshift-default-8dcc78969-vk4tp"]
Nov 01 13:02:32.189583 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:32.189553    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" path="/var/lib/kubelet/pods/94d15666-ddd7-4270-8e31-4630a4b89b39/volumes"
Nov 01 13:02:35.823820 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:35.823784    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"]
Nov 01 13:02:35.824315 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:35.824000    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy" containerID="cri-o://e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7" gracePeriod=30
Nov 01 13:02:35.851068 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:35.851039    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"]
Nov 01 13:02:35.851256 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:35.851235    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy" containerID="cri-o://53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350" gracePeriod=30
Nov 01 13:02:35.863703 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:35.863677    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"]
Nov 01 13:02:35.863869 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:35.863850    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy" containerID="cri-o://ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c" gracePeriod=30
Nov 01 13:02:35.872363 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:35.872335    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 13:02:35.872470 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:35.872401    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 13:02:37.221981 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:37.221945    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 13:02:37.222435 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:37.222016    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 13:02:37.233715 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:37.233685    4095 patch_prober.go:29] interesting pod/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 13:02:37.233809 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:37.233737    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 13:02:41.124830 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.124792    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:02:41.155355 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.155327    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:02:41.171291 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.171267    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:02:41.171851 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.171831    4095 generic.go:356] "Generic (PLEG): container finished" podID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerID="ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c" exitCode=0
Nov 01 13:02:41.171933 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.171902    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" event={"ID":"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a","Type":"ContainerDied","Data":"ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c"}
Nov 01 13:02:41.171994 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.171955    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd" event={"ID":"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a","Type":"ContainerDied","Data":"f93dfc6a8100a876c17651eb321c854ea3a829270615ef621cfb3cb8fba65622"}
Nov 01 13:02:41.172036 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.172013    4095 scope.go:122] "RemoveContainer" containerID="ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c"
Nov 01 13:02:41.173139 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.173117    4095 generic.go:356] "Generic (PLEG): container finished" podID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerID="53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350" exitCode=0
Nov 01 13:02:41.173246 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.173162    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"
Nov 01 13:02:41.173246 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.173174    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" event={"ID":"4816ead2-e1e6-4316-968a-e1e2ea46093c","Type":"ContainerDied","Data":"53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350"}
Nov 01 13:02:41.173246 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.173225    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x" event={"ID":"4816ead2-e1e6-4316-968a-e1e2ea46093c","Type":"ContainerDied","Data":"0feb7fd7d991fbec24913479ac9e4e0aae687d37cc51c066024cf52bfba107b7"}
Nov 01 13:02:41.174689 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.174670    4095 generic.go:356] "Generic (PLEG): container finished" podID="c6735977-122d-4a0f-b5a8-b200856623e1" containerID="e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7" exitCode=0
Nov 01 13:02:41.174773 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.174700    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" event={"ID":"c6735977-122d-4a0f-b5a8-b200856623e1","Type":"ContainerDied","Data":"e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7"}
Nov 01 13:02:41.174773 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.174731    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"
Nov 01 13:02:41.174773 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.174739    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp" event={"ID":"c6735977-122d-4a0f-b5a8-b200856623e1","Type":"ContainerDied","Data":"3ec5408aced1c2d3c4e74fbf80d145415b8fc65d9042507538dd827480acdff8"}
Nov 01 13:02:41.181191 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.181175    4095 scope.go:122] "RemoveContainer" containerID="ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c"
Nov 01 13:02:41.181469 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:02:41.181449    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c\": container with ID starting with ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c not found: ID does not exist" containerID="ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c"
Nov 01 13:02:41.181568 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.181473    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c"} err="failed to get container status \"ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c\": rpc error: code = NotFound desc = could not find container \"ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c\": container with ID starting with ede007a63d7bcbbaf1931acd599841d6ef90c59420130d89d512862b9a45d41c not found: ID does not exist"
Nov 01 13:02:41.181568 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.181495    4095 scope.go:122] "RemoveContainer" containerID="53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350"
Nov 01 13:02:41.189580 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.189563    4095 scope.go:122] "RemoveContainer" containerID="53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350"
Nov 01 13:02:41.189824 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:02:41.189808    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350\": container with ID starting with 53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350 not found: ID does not exist" containerID="53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350"
Nov 01 13:02:41.189875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.189839    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350"} err="failed to get container status \"53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350\": rpc error: code = NotFound desc = could not find container \"53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350\": container with ID starting with 53ff0f9454aea7502613456d8d74aca65c88f734f72a62225968cfe6447c6350 not found: ID does not exist"
Nov 01 13:02:41.189875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.189853    4095 scope.go:122] "RemoveContainer" containerID="e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7"
Nov 01 13:02:41.197674 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.197659    4095 scope.go:122] "RemoveContainer" containerID="e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7"
Nov 01 13:02:41.197933 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:02:41.197915    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7\": container with ID starting with e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7 not found: ID does not exist" containerID="e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7"
Nov 01 13:02:41.198013 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.197938    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7"} err="failed to get container status \"e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7\": rpc error: code = NotFound desc = could not find container \"e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7\": container with ID starting with e87c8f4d96a771b53db0a41fa61cbbcf68396fd0dbf0077647311f4d156b94a7 not found: ID does not exist"
Nov 01 13:02:41.240528 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240501    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-certs\") pod \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") "
Nov 01 13:02:41.240670 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240550    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-certs\") pod \"4816ead2-e1e6-4316-968a-e1e2ea46093c\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") "
Nov 01 13:02:41.240670 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240587    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6735977-122d-4a0f-b5a8-b200856623e1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6735977-122d-4a0f-b5a8-b200856623e1-istiod-ca-cert\") pod \"c6735977-122d-4a0f-b5a8-b200856623e1\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") "
Nov 01 13:02:41.240670 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240615    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4816ead2-e1e6-4316-968a-e1e2ea46093c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4816ead2-e1e6-4316-968a-e1e2ea46093c-istiod-ca-cert\") pod \"4816ead2-e1e6-4316-968a-e1e2ea46093c\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") "
Nov 01 13:02:41.240670 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240639    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-kube-api-access-68srf\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-kube-api-access-68srf\") pod \"c6735977-122d-4a0f-b5a8-b200856623e1\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") "
Nov 01 13:02:41.240893 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240797    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-socket\") pod \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") "
Nov 01 13:02:41.240893 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240837    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-kube-api-access-tjwv5\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-kube-api-access-tjwv5\") pod \"4816ead2-e1e6-4316-968a-e1e2ea46093c\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") "
Nov 01 13:02:41.240893 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240845    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-certs" pod "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" (UID: "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.240893 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240880    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-certs" pod "4816ead2-e1e6-4316-968a-e1e2ea46093c" (UID: "4816ead2-e1e6-4316-968a-e1e2ea46093c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.240893 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240890    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-credential-socket\") pod \"c6735977-122d-4a0f-b5a8-b200856623e1\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") "
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.240921    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-certs\") pod \"c6735977-122d-4a0f-b5a8-b200856623e1\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") "
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241003    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-podinfo\") pod \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") "
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241018    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6735977-122d-4a0f-b5a8-b200856623e1-istiod-ca-cert" pod "c6735977-122d-4a0f-b5a8-b200856623e1" (UID: "c6735977-122d-4a0f-b5a8-b200856623e1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241040    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-kube-api-access-8hdvj\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-kube-api-access-8hdvj\") pod \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") "
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241066    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-socket\") pod \"4816ead2-e1e6-4316-968a-e1e2ea46093c\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") "
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241040    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4816ead2-e1e6-4316-968a-e1e2ea46093c-istiod-ca-cert" pod "4816ead2-e1e6-4316-968a-e1e2ea46093c" (UID: "4816ead2-e1e6-4316-968a-e1e2ea46093c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241130    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-credential-socket\") pod \"4816ead2-e1e6-4316-968a-e1e2ea46093c\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") "
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241157    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-credential-socket" pod "c6735977-122d-4a0f-b5a8-b200856623e1" (UID: "c6735977-122d-4a0f-b5a8-b200856623e1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241173    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istiod-ca-cert\") pod \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") "
Nov 01 13:02:41.241206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241199    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-envoy\") pod \"4816ead2-e1e6-4316-968a-e1e2ea46093c\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241235    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-podinfo\") pod \"4816ead2-e1e6-4316-968a-e1e2ea46093c\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241271    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-envoy\") pod \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241313    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-token\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-token\") pod \"4816ead2-e1e6-4316-968a-e1e2ea46093c\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241339    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-certs" pod "c6735977-122d-4a0f-b5a8-b200856623e1" (UID: "c6735977-122d-4a0f-b5a8-b200856623e1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241352    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-socket" pod "4816ead2-e1e6-4316-968a-e1e2ea46093c" (UID: "4816ead2-e1e6-4316-968a-e1e2ea46093c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241345    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-data\") pod \"c6735977-122d-4a0f-b5a8-b200856623e1\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241408    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-data\") pod \"4816ead2-e1e6-4316-968a-e1e2ea46093c\" (UID: \"4816ead2-e1e6-4316-968a-e1e2ea46093c\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241453    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-socket" pod "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" (UID: "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241470    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-token\") pod \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241525    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-credential-socket\") pod \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241560    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-data" pod "c6735977-122d-4a0f-b5a8-b200856623e1" (UID: "c6735977-122d-4a0f-b5a8-b200856623e1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241564    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6735977-122d-4a0f-b5a8-b200856623e1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6735977-122d-4a0f-b5a8-b200856623e1-istio-podinfo\") pod \"c6735977-122d-4a0f-b5a8-b200856623e1\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241652    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-istio-token\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-istio-token\") pod \"c6735977-122d-4a0f-b5a8-b200856623e1\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241671    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-data" pod "4816ead2-e1e6-4316-968a-e1e2ea46093c" (UID: "4816ead2-e1e6-4316-968a-e1e2ea46093c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241695    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-envoy\") pod \"c6735977-122d-4a0f-b5a8-b200856623e1\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") "
Nov 01 13:02:41.241723 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241723    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-data\") pod \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\" (UID: \"ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a\") "
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241757    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-socket\") pod \"c6735977-122d-4a0f-b5a8-b200856623e1\" (UID: \"c6735977-122d-4a0f-b5a8-b200856623e1\") "
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.241863    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-credential-socket" pod "4816ead2-e1e6-4316-968a-e1e2ea46093c" (UID: "4816ead2-e1e6-4316-968a-e1e2ea46093c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242476    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242499    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242514    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242530    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242545    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6735977-122d-4a0f-b5a8-b200856623e1-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242563    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4816ead2-e1e6-4316-968a-e1e2ea46093c-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242578    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242594    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.242613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242608    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.243278 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242622    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.243278 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.242637    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.243397 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.243294    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istiod-ca-cert" pod "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" (UID: "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:02:41.243397 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.243387    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-credential-socket" pod "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" (UID: "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.243702 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.243679    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-data" pod "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" (UID: "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.243929 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.243601    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-socket" pod "c6735977-122d-4a0f-b5a8-b200856623e1" (UID: "c6735977-122d-4a0f-b5a8-b200856623e1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.244057 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.244029    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-kube-api-access-68srf" pod "c6735977-122d-4a0f-b5a8-b200856623e1" (UID: "c6735977-122d-4a0f-b5a8-b200856623e1"). InnerVolumeSpecName "kube-api-access-68srf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.244168 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.244097    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-envoy" pod "4816ead2-e1e6-4316-968a-e1e2ea46093c" (UID: "4816ead2-e1e6-4316-968a-e1e2ea46093c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.244168 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.244099    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-kube-api-access-tjwv5" pod "4816ead2-e1e6-4316-968a-e1e2ea46093c" (UID: "4816ead2-e1e6-4316-968a-e1e2ea46093c"). InnerVolumeSpecName "kube-api-access-tjwv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.244296 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.244218    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6735977-122d-4a0f-b5a8-b200856623e1-istio-podinfo" pod "c6735977-122d-4a0f-b5a8-b200856623e1" (UID: "c6735977-122d-4a0f-b5a8-b200856623e1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:02:41.244493 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.244470    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-podinfo" pod "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" (UID: "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:02:41.245191 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.245126    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-kube-api-access-8hdvj" pod "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" (UID: "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a"). InnerVolumeSpecName "kube-api-access-8hdvj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.245293 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.245248    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-token" pod "4816ead2-e1e6-4316-968a-e1e2ea46093c" (UID: "4816ead2-e1e6-4316-968a-e1e2ea46093c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.245619 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.245599    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-podinfo" pod "4816ead2-e1e6-4316-968a-e1e2ea46093c" (UID: "4816ead2-e1e6-4316-968a-e1e2ea46093c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:02:41.245775 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.245760    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-istio-token" pod "c6735977-122d-4a0f-b5a8-b200856623e1" (UID: "c6735977-122d-4a0f-b5a8-b200856623e1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.246266 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.246242    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-envoy" pod "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" (UID: "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.246266 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.246252    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-token" pod "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" (UID: "ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:02:41.246380 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.246319    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-envoy" pod "c6735977-122d-4a0f-b5a8-b200856623e1" (UID: "c6735977-122d-4a0f-b5a8-b200856623e1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:02:41.343513 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343474    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343513 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343501    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hdvj\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-kube-api-access-8hdvj\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343522    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343534    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343546    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343560    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343571    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343583    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343595    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343608    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6735977-122d-4a0f-b5a8-b200856623e1-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343622    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343635    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343647    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343658    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6735977-122d-4a0f-b5a8-b200856623e1-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343670    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68srf\" (UniqueName: \"kubernetes.io/projected/c6735977-122d-4a0f-b5a8-b200856623e1-kube-api-access-68srf\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.343752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.343683    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjwv5\" (UniqueName: \"kubernetes.io/projected/4816ead2-e1e6-4316-968a-e1e2ea46093c-kube-api-access-tjwv5\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:02:41.663210 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.663176    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"]
Nov 01 13:02:41.668488 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.668459    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-fqc8x"]
Nov 01 13:02:41.774700 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.774614    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"]
Nov 01 13:02:41.778600 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:41.778570    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-wnflp"]
Nov 01 13:02:42.180640 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:42.180610    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"
Nov 01 13:02:42.189557 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:42.189526    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" path="/var/lib/kubelet/pods/4816ead2-e1e6-4316-968a-e1e2ea46093c/volumes"
Nov 01 13:02:42.190272 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:42.190190    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" path="/var/lib/kubelet/pods/c6735977-122d-4a0f-b5a8-b200856623e1/volumes"
Nov 01 13:02:42.336429 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:42.336397    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"]
Nov 01 13:02:42.340061 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:42.340037    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-187d06ed-openshift-default-544c4b9b45-k48wd"]
Nov 01 13:02:44.188696 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:44.188652    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" path="/var/lib/kubelet/pods/ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a/volumes"
Nov 01 13:02:49.114054 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:02:49.114010    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:02:54.598938 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.598897    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"]
Nov 01 13:02:54.599496 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599464    4095 cpu_manager.go:414] "containerMap: removing container" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy"
Nov 01 13:02:54.599496 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599482    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy"
Nov 01 13:02:54.599573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599508    4095 cpu_manager.go:414] "containerMap: removing container" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy"
Nov 01 13:02:54.599573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599513    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy"
Nov 01 13:02:54.599573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599528    4095 cpu_manager.go:414] "containerMap: removing container" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy"
Nov 01 13:02:54.599573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599533    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy"
Nov 01 13:02:54.599573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599546    4095 cpu_manager.go:414] "containerMap: removing container" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy"
Nov 01 13:02:54.599573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599551    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy"
Nov 01 13:02:54.599573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599563    4095 cpu_manager.go:414] "containerMap: removing container" podUID="18010a26-70d2-46fb-a44e-3bedc5c1d6b1" containerName="maas-api"
Nov 01 13:02:54.599573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599568    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18010a26-70d2-46fb-a44e-3bedc5c1d6b1" containerName="maas-api"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599578    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2fdbf1d4-410e-4731-a492-3c425e5090e9" containerName="revoke-keys"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599582    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fdbf1d4-410e-4731-a492-3c425e5090e9" containerName="revoke-keys"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599592    4095 cpu_manager.go:414] "containerMap: removing container" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599597    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599673    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6735977-122d-4a0f-b5a8-b200856623e1" containerName="istio-proxy"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599684    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="18010a26-70d2-46fb-a44e-3bedc5c1d6b1" containerName="maas-api"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599693    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fdbf1d4-410e-4731-a492-3c425e5090e9" containerName="revoke-keys"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599703    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="4816ead2-e1e6-4316-968a-e1e2ea46093c" containerName="istio-proxy"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599713    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599720    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="ccc8a7e4-94f6-4990-a8f1-7d78bcf89e2a" containerName="istio-proxy"
Nov 01 13:02:54.599826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.599730    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="94d15666-ddd7-4270-8e31-4630a4b89b39" containerName="istio-proxy"
Nov 01 13:02:54.603246 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.603228    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.611226 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.611204    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.611305 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.611238    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.611349 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.611319    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.611349 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.611345    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.611441 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.611360    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3566191-2934-4500-90e2-c12dbffb0354-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.611517 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.611496    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2kx\" (UniqueName: \"kubernetes.io/projected/f3566191-2934-4500-90e2-c12dbffb0354-kube-api-access-rd2kx\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.611561 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.611544    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.611679 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.611658    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3566191-2934-4500-90e2-c12dbffb0354-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.611759 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.611744    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3566191-2934-4500-90e2-c12dbffb0354-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713044 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713001    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713044 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713050    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713323 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713084    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713323 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713105    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713323 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713122    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3566191-2934-4500-90e2-c12dbffb0354-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713323 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713186    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2kx\" (UniqueName: \"kubernetes.io/projected/f3566191-2934-4500-90e2-c12dbffb0354-kube-api-access-rd2kx\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713323 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713256    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713323 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713311    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3566191-2934-4500-90e2-c12dbffb0354-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713698 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713389    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3566191-2934-4500-90e2-c12dbffb0354-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713698 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713494    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713816 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713719    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.713816 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.713773    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.714081 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.714056    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.714327 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.714297    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3566191-2934-4500-90e2-c12dbffb0354-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.715889 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.715870    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3566191-2934-4500-90e2-c12dbffb0354-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.716128 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.716110    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3566191-2934-4500-90e2-c12dbffb0354-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.767180 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.767158    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2kx\" (UniqueName: \"kubernetes.io/projected/f3566191-2934-4500-90e2-c12dbffb0354-kube-api-access-rd2kx\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.767180 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.767179    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3566191-2934-4500-90e2-c12dbffb0354-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-9m98v\" (UID: \"f3566191-2934-4500-90e2-c12dbffb0354\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:54.789232 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.789192    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"]
Nov 01 13:02:54.916027 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:54.915988    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:55.340883 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:55.340797    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"]
Nov 01 13:02:55.346294 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:02:55.346263    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3566191_2934_4500_90e2_c12dbffb0354.slice/crio-2d9be6ef93c6f06da8aa621d6a77c64f3043cdc3c53380dfb945f8ba6da3a741 WatchSource:0}: Error finding container 2d9be6ef93c6f06da8aa621d6a77c64f3043cdc3c53380dfb945f8ba6da3a741: Status 404 returned error can't find the container with id 2d9be6ef93c6f06da8aa621d6a77c64f3043cdc3c53380dfb945f8ba6da3a741
Nov 01 13:02:55.348533 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:55.348498    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:02:55.348811 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:55.348572    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:02:55.348811 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:55.348610    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:02:56.300514 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:56.300480    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" event={"ID":"f3566191-2934-4500-90e2-c12dbffb0354","Type":"ContainerStarted","Data":"edefc8217bec2885a48cb8e626f896c33ba6928208e10b2c79752b0686c5fbca"}
Nov 01 13:02:56.300945 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:56.300531    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" event={"ID":"f3566191-2934-4500-90e2-c12dbffb0354","Type":"ContainerStarted","Data":"2d9be6ef93c6f06da8aa621d6a77c64f3043cdc3c53380dfb945f8ba6da3a741"}
Nov 01 13:02:56.463171 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:56.463116    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" podStartSLOduration=2.463101451 podStartE2EDuration="2.463101451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:02:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:02:56.460838601 +0000 UTC m=+2436.378634281" watchObservedRunningTime="2026-11-01 13:02:56.463101451 +0000 UTC m=+2436.380897187"
Nov 01 13:02:56.924125 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:56.924086    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-9m98v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 13:02:56.924431 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:56.924403    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" podUID="f3566191-2934-4500-90e2-c12dbffb0354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 13:02:56.924866 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:56.924208    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:02:57.924014 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:57.922419    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-9m98v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 13:02:57.924014 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:57.922493    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" podUID="f3566191-2934-4500-90e2-c12dbffb0354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 13:02:58.921964 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:58.921469    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-9m98v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 13:02:58.921964 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:58.921553    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" podUID="f3566191-2934-4500-90e2-c12dbffb0354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 13:02:59.916368 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:59.916332    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-9m98v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 13:02:59.917033 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:02:59.916956    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" podUID="f3566191-2934-4500-90e2-c12dbffb0354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 13:03:00.917441 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:00.917403    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-9m98v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 13:03:00.918065 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:00.918032    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" podUID="f3566191-2934-4500-90e2-c12dbffb0354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 13:03:01.916492 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:01.916457    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-9m98v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 13:03:01.916689 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:01.916511    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" podUID="f3566191-2934-4500-90e2-c12dbffb0354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 13:03:02.916892 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:02.916844    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-9m98v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body=
Nov 01 13:03:02.917337 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:02.916923    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" podUID="f3566191-2934-4500-90e2-c12dbffb0354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused"
Nov 01 13:03:03.921091 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:03.921058    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:03:03.921580 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:03.921437    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:03:03.922146 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:03.922126    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v"
Nov 01 13:03:13.613630 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.613586    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"]
Nov 01 13:03:13.617799 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.617783    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.672299 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.672263    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"]
Nov 01 13:03:13.715981 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.715934    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:03:13.716227 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.716207    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:03:13.729189 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.729164    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52b3c667-krgfd\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.729319 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.729217    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26njc\" (UniqueName: \"kubernetes.io/projected/27d8fb21-1d01-453a-b79c-16a851bfb052-kube-api-access-26njc\") pod \"maas-api-revoke-sub-52b3c667-krgfd\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.729378 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.729359    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52b3c667-krgfd\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.830768 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.830741    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52b3c667-krgfd\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.830930 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.830808    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52b3c667-krgfd\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.830930 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.830845    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26njc\" (UniqueName: \"kubernetes.io/projected/27d8fb21-1d01-453a-b79c-16a851bfb052-kube-api-access-26njc\") pod \"maas-api-revoke-sub-52b3c667-krgfd\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.831432 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.831409    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-52b3c667-krgfd\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.833124 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.833101    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-xks-ca\") pod \"maas-api-revoke-sub-52b3c667-krgfd\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.885323 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.885266    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26njc\" (UniqueName: \"kubernetes.io/projected/27d8fb21-1d01-453a-b79c-16a851bfb052-kube-api-access-26njc\") pod \"maas-api-revoke-sub-52b3c667-krgfd\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:13.931659 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:13.931627    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:14.297075 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:14.297049    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"]
Nov 01 13:03:14.297917 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:03:14.297885    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27d8fb21_1d01_453a_b79c_16a851bfb052.slice/crio-4bd15b3f08b3153db9fdbc9693c9754e2b89c5d442e9dc9109bc45a1e29f00d6 WatchSource:0}: Error finding container 4bd15b3f08b3153db9fdbc9693c9754e2b89c5d442e9dc9109bc45a1e29f00d6: Status 404 returned error can't find the container with id 4bd15b3f08b3153db9fdbc9693c9754e2b89c5d442e9dc9109bc45a1e29f00d6
Nov 01 13:03:14.428720 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:14.428684    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd" event={"ID":"27d8fb21-1d01-453a-b79c-16a851bfb052","Type":"ContainerStarted","Data":"4bd15b3f08b3153db9fdbc9693c9754e2b89c5d442e9dc9109bc45a1e29f00d6"}
Nov 01 13:03:15.435735 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:15.435651    4095 generic.go:356] "Generic (PLEG): container finished" podID="27d8fb21-1d01-453a-b79c-16a851bfb052" containerID="f7f4afb28b31a16ae3da00b50036b8cb53840a1cb6b339982ec73ca8569bf8bd" exitCode=0
Nov 01 13:03:15.436133 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:15.435734    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd" event={"ID":"27d8fb21-1d01-453a-b79c-16a851bfb052","Type":"ContainerDied","Data":"f7f4afb28b31a16ae3da00b50036b8cb53840a1cb6b339982ec73ca8569bf8bd"}
Nov 01 13:03:16.586049 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.586020    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:16.665381 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.665339    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-openshift-ca\") pod \"27d8fb21-1d01-453a-b79c-16a851bfb052\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") "
Nov 01 13:03:16.665532 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.665464    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-xks-ca\") pod \"27d8fb21-1d01-453a-b79c-16a851bfb052\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") "
Nov 01 13:03:16.665593 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.665528    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27d8fb21-1d01-453a-b79c-16a851bfb052-kube-api-access-26njc\" (UniqueName: \"kubernetes.io/projected/27d8fb21-1d01-453a-b79c-16a851bfb052-kube-api-access-26njc\") pod \"27d8fb21-1d01-453a-b79c-16a851bfb052\" (UID: \"27d8fb21-1d01-453a-b79c-16a851bfb052\") "
Nov 01 13:03:16.665749 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.665729    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-openshift-ca" pod "27d8fb21-1d01-453a-b79c-16a851bfb052" (UID: "27d8fb21-1d01-453a-b79c-16a851bfb052"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:03:16.665932 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.665916    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-openshift-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:16.667502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.667479    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27d8fb21-1d01-453a-b79c-16a851bfb052-kube-api-access-26njc" pod "27d8fb21-1d01-453a-b79c-16a851bfb052" (UID: "27d8fb21-1d01-453a-b79c-16a851bfb052"). InnerVolumeSpecName "kube-api-access-26njc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:03:16.667502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.667484    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-xks-ca" pod "27d8fb21-1d01-453a-b79c-16a851bfb052" (UID: "27d8fb21-1d01-453a-b79c-16a851bfb052"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:03:16.766994 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.766880    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/27d8fb21-1d01-453a-b79c-16a851bfb052-maas-api-xks-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:16.766994 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:16.766913    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26njc\" (UniqueName: \"kubernetes.io/projected/27d8fb21-1d01-453a-b79c-16a851bfb052-kube-api-access-26njc\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:17.449263 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:17.449232    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"
Nov 01 13:03:17.449447 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:17.449263    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd" event={"ID":"27d8fb21-1d01-453a-b79c-16a851bfb052","Type":"ContainerDied","Data":"4bd15b3f08b3153db9fdbc9693c9754e2b89c5d442e9dc9109bc45a1e29f00d6"}
Nov 01 13:03:17.449447 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:17.449317    4095 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4bd15b3f08b3153db9fdbc9693c9754e2b89c5d442e9dc9109bc45a1e29f00d6"
Nov 01 13:03:19.115856 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:03:19.115821    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:03:24.963905 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:24.963870    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"]
Nov 01 13:03:24.964506 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:24.964484    4095 cpu_manager.go:414] "containerMap: removing container" podUID="27d8fb21-1d01-453a-b79c-16a851bfb052" containerName="revoke-keys"
Nov 01 13:03:24.964554 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:24.964508    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27d8fb21-1d01-453a-b79c-16a851bfb052" containerName="revoke-keys"
Nov 01 13:03:24.964611 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:24.964600    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="27d8fb21-1d01-453a-b79c-16a851bfb052" containerName="revoke-keys"
Nov 01 13:03:24.969276 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:24.969258    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.019191 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.019172    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:03:25.055379 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.055352    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.055539 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.055413    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6j86l\" (UniqueName: \"kubernetes.io/projected/db4916da-46d6-4c6f-b970-0f504b3efb56-kube-api-access-6j86l\") pod \"maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.055622 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.055561    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.066653 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.066619    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:03:25.069172 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.069153    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"]
Nov 01 13:03:25.156762 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.156736    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.156905 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.156781    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6j86l\" (UniqueName: \"kubernetes.io/projected/db4916da-46d6-4c6f-b970-0f504b3efb56-kube-api-access-6j86l\") pod \"maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.156905 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.156838    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.157491 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.157464    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.159108 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.159087    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.211630 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.211585    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6j86l\" (UniqueName: \"kubernetes.io/projected/db4916da-46d6-4c6f-b970-0f504b3efb56-kube-api-access-6j86l\") pod \"maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.283834 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.283752    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:25.654450 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:25.654415    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"]
Nov 01 13:03:25.655645 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:03:25.655615    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb4916da_46d6_4c6f_b970_0f504b3efb56.slice/crio-bb1d412dbae1ba2acaba18c23131e1aaee456afa0e5008374da8347af29a0780 WatchSource:0}: Error finding container bb1d412dbae1ba2acaba18c23131e1aaee456afa0e5008374da8347af29a0780: Status 404 returned error can't find the container with id bb1d412dbae1ba2acaba18c23131e1aaee456afa0e5008374da8347af29a0780
Nov 01 13:03:26.500892 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:26.500863    4095 generic.go:356] "Generic (PLEG): container finished" podID="db4916da-46d6-4c6f-b970-0f504b3efb56" containerID="3ec07766034f918d01987afe8274f134e6c751d5045376060e51356b70a0e7d8" exitCode=0
Nov 01 13:03:26.501306 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:26.500944    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g" event={"ID":"db4916da-46d6-4c6f-b970-0f504b3efb56","Type":"ContainerDied","Data":"3ec07766034f918d01987afe8274f134e6c751d5045376060e51356b70a0e7d8"}
Nov 01 13:03:26.501306 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:26.501038    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g" event={"ID":"db4916da-46d6-4c6f-b970-0f504b3efb56","Type":"ContainerStarted","Data":"bb1d412dbae1ba2acaba18c23131e1aaee456afa0e5008374da8347af29a0780"}
Nov 01 13:03:27.651939 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.651915    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:27.683998 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.683959    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-xks-ca\") pod \"db4916da-46d6-4c6f-b970-0f504b3efb56\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") "
Nov 01 13:03:27.684121 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.684080    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db4916da-46d6-4c6f-b970-0f504b3efb56-kube-api-access-6j86l\" (UniqueName: \"kubernetes.io/projected/db4916da-46d6-4c6f-b970-0f504b3efb56-kube-api-access-6j86l\") pod \"db4916da-46d6-4c6f-b970-0f504b3efb56\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") "
Nov 01 13:03:27.684207 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.684194    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-openshift-ca\") pod \"db4916da-46d6-4c6f-b970-0f504b3efb56\" (UID: \"db4916da-46d6-4c6f-b970-0f504b3efb56\") "
Nov 01 13:03:27.684517 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.684491    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-openshift-ca" pod "db4916da-46d6-4c6f-b970-0f504b3efb56" (UID: "db4916da-46d6-4c6f-b970-0f504b3efb56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:03:27.686079 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.686058    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db4916da-46d6-4c6f-b970-0f504b3efb56-kube-api-access-6j86l" pod "db4916da-46d6-4c6f-b970-0f504b3efb56" (UID: "db4916da-46d6-4c6f-b970-0f504b3efb56"). InnerVolumeSpecName "kube-api-access-6j86l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:03:27.686172 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.686059    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-xks-ca" pod "db4916da-46d6-4c6f-b970-0f504b3efb56" (UID: "db4916da-46d6-4c6f-b970-0f504b3efb56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:03:27.785613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.785527    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6j86l\" (UniqueName: \"kubernetes.io/projected/db4916da-46d6-4c6f-b970-0f504b3efb56-kube-api-access-6j86l\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:27.785613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.785560    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-openshift-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:27.785613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:27.785577    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/db4916da-46d6-4c6f-b970-0f504b3efb56-maas-api-xks-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:28.513157 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:28.513076    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"
Nov 01 13:03:28.513157 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:28.513092    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g" event={"ID":"db4916da-46d6-4c6f-b970-0f504b3efb56","Type":"ContainerDied","Data":"bb1d412dbae1ba2acaba18c23131e1aaee456afa0e5008374da8347af29a0780"}
Nov 01 13:03:28.513157 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:28.513141    4095 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb1d412dbae1ba2acaba18c23131e1aaee456afa0e5008374da8347af29a0780"
Nov 01 13:03:29.289385 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:29.289354    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"]
Nov 01 13:03:44.310160 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.310055    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main" containerID="cri-o://b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25" gracePeriod=45
Nov 01 13:03:44.586397 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.586372    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 13:03:44.625296 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.625266    4095 generic.go:356] "Generic (PLEG): container finished" podID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerID="b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25" exitCode=0
Nov 01 13:03:44.625421 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.625322    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"
Nov 01 13:03:44.625421 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.625354    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" event={"ID":"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012","Type":"ContainerDied","Data":"b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25"}
Nov 01 13:03:44.625421 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.625397    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6" event={"ID":"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012","Type":"ContainerDied","Data":"d9f6cfe1d3ea7c02406aed62bff9feea5e99605874c3963928bd9414454ece14"}
Nov 01 13:03:44.625421 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.625415    4095 scope.go:122] "RemoveContainer" containerID="b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25"
Nov 01 13:03:44.634246 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.634230    4095 scope.go:122] "RemoveContainer" containerID="b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25"
Nov 01 13:03:44.634504 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:03:44.634485    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25\": container with ID starting with b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25 not found: ID does not exist" containerID="b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25"
Nov 01 13:03:44.634555 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.634518    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25"} err="failed to get container status \"b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25\": rpc error: code = NotFound desc = could not find container \"b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25\": container with ID starting with b7b495179cce855a94c53bc70a47324472b24ed0d9bbdb014328ec99e201eb25 not found: ID does not exist"
Nov 01 13:03:44.677979 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.677941    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-kube-api-access-xfnnr\" (UniqueName: \"kubernetes.io/projected/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-kube-api-access-xfnnr\") pod \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") "
Nov 01 13:03:44.678136 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678003    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tls-certs\" (UniqueName: \"kubernetes.io/secret/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tls-certs\") pod \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") "
Nov 01 13:03:44.678136 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678085    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tmp-dir\") pod \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") "
Nov 01 13:03:44.678136 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678110    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-model-cache\") pod \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") "
Nov 01 13:03:44.678321 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678133    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-dshm\") pod \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") "
Nov 01 13:03:44.678321 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678163    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-home\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-home\") pod \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\" (UID: \"d60a0271-a7d9-4a8c-8bd3-34ad08fb0012\") "
Nov 01 13:03:44.678444 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678417    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tmp-dir" pod "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" (UID: "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:03:44.678444 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678423    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-model-cache" pod "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" (UID: "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:03:44.678573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678529    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-home" pod "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" (UID: "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:03:44.678669 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678654    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:44.678735 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678671    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:44.678735 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.678681    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:44.680320 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.680297    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tls-certs" pod "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" (UID: "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:03:44.680414 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.680324    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-dshm" pod "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" (UID: "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:03:44.680414 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.680395    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-kube-api-access-xfnnr" pod "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" (UID: "d60a0271-a7d9-4a8c-8bd3-34ad08fb0012"). InnerVolumeSpecName "kube-api-access-xfnnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:03:44.780332 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.780299    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:44.780332 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.780327    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfnnr\" (UniqueName: \"kubernetes.io/projected/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-kube-api-access-xfnnr\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:44.780332 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.780338    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:44.980851 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.980820    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"]
Nov 01 13:03:44.991677 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.991639    4095 cpu_manager.go:414] "containerMap: removing container" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main"
Nov 01 13:03:44.991677 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.991669    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main"
Nov 01 13:03:44.991904 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.991739    4095 cpu_manager.go:414] "containerMap: removing container" podUID="db4916da-46d6-4c6f-b970-0f504b3efb56" containerName="revoke-keys"
Nov 01 13:03:44.991904 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.991749    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db4916da-46d6-4c6f-b970-0f504b3efb56" containerName="revoke-keys"
Nov 01 13:03:44.992157 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.992136    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" containerName="main"
Nov 01 13:03:44.992298 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.992164    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="db4916da-46d6-4c6f-b970-0f504b3efb56" containerName="revoke-keys"
Nov 01 13:03:44.998207 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:44.998183    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.038139 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.038108    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"]
Nov 01 13:03:45.056090 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.055888    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:03:45.056261 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.056208    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:03:45.058424 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.058405    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"]
Nov 01 13:03:45.063639 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.063617    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-ecd640/distinct-2-w3-ecd640-kserve-84d797b4c-295q6"]
Nov 01 13:03:45.084901 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.084878    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.085054 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.084917    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kl9p2\" (UniqueName: \"kubernetes.io/projected/1e0f98b9-19d3-4070-b275-a8a9620f242e-kube-api-access-kl9p2\") pod \"maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.085148 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.085133    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.186024 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.185932    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.186024 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.186004    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kl9p2\" (UniqueName: \"kubernetes.io/projected/1e0f98b9-19d3-4070-b275-a8a9620f242e-kube-api-access-kl9p2\") pod \"maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.186248 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.186114    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.186622 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.186596    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.188360 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.188342    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.242253 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.242229    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kl9p2\" (UniqueName: \"kubernetes.io/projected/1e0f98b9-19d3-4070-b275-a8a9620f242e-kube-api-access-kl9p2\") pod \"maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.311581 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.311551    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:45.687022 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:45.686994    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"]
Nov 01 13:03:45.688565 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:03:45.688542    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0f98b9_19d3_4070_b275_a8a9620f242e.slice/crio-c6af512bd36fa6b347a4ec34eec42c4fd3b79c0f683eb012499d11aaa47d618c WatchSource:0}: Error finding container c6af512bd36fa6b347a4ec34eec42c4fd3b79c0f683eb012499d11aaa47d618c: Status 404 returned error can't find the container with id c6af512bd36fa6b347a4ec34eec42c4fd3b79c0f683eb012499d11aaa47d618c
Nov 01 13:03:46.188262 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:46.188221    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d60a0271-a7d9-4a8c-8bd3-34ad08fb0012" path="/var/lib/kubelet/pods/d60a0271-a7d9-4a8c-8bd3-34ad08fb0012/volumes"
Nov 01 13:03:46.638498 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:46.638463    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5" event={"ID":"1e0f98b9-19d3-4070-b275-a8a9620f242e","Type":"ContainerStarted","Data":"f10ec798419664baaddfce2077f7182c96f5aefc85b49b8a4b2f702fb17cb3d4"}
Nov 01 13:03:46.638902 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:46.638516    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5" event={"ID":"1e0f98b9-19d3-4070-b275-a8a9620f242e","Type":"ContainerStarted","Data":"c6af512bd36fa6b347a4ec34eec42c4fd3b79c0f683eb012499d11aaa47d618c"}
Nov 01 13:03:46.775447 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:46.775395    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5" podStartSLOduration=2.7753807569999998 podStartE2EDuration="2.775380757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:03:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:03:46.77431959 +0000 UTC m=+2486.692115282" watchObservedRunningTime="2026-11-01 13:03:46.775380757 +0000 UTC m=+2486.693176422"
Nov 01 13:03:47.646804 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:47.646769    4095 generic.go:356] "Generic (PLEG): container finished" podID="1e0f98b9-19d3-4070-b275-a8a9620f242e" containerID="f10ec798419664baaddfce2077f7182c96f5aefc85b49b8a4b2f702fb17cb3d4" exitCode=0
Nov 01 13:03:47.647249 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:47.646861    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5" event={"ID":"1e0f98b9-19d3-4070-b275-a8a9620f242e","Type":"ContainerDied","Data":"f10ec798419664baaddfce2077f7182c96f5aefc85b49b8a4b2f702fb17cb3d4"}
Nov 01 13:03:48.792418 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:48.792396    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:48.927685 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:48.927605    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-openshift-ca\") pod \"1e0f98b9-19d3-4070-b275-a8a9620f242e\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") "
Nov 01 13:03:48.927845 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:48.927764    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-xks-ca\") pod \"1e0f98b9-19d3-4070-b275-a8a9620f242e\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") "
Nov 01 13:03:48.927915 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:48.927853    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0f98b9-19d3-4070-b275-a8a9620f242e-kube-api-access-kl9p2\" (UniqueName: \"kubernetes.io/projected/1e0f98b9-19d3-4070-b275-a8a9620f242e-kube-api-access-kl9p2\") pod \"1e0f98b9-19d3-4070-b275-a8a9620f242e\" (UID: \"1e0f98b9-19d3-4070-b275-a8a9620f242e\") "
Nov 01 13:03:48.928055 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:48.928026    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-openshift-ca" pod "1e0f98b9-19d3-4070-b275-a8a9620f242e" (UID: "1e0f98b9-19d3-4070-b275-a8a9620f242e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:03:48.928312 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:48.928291    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-openshift-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:48.930641 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:48.930603    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-xks-ca" pod "1e0f98b9-19d3-4070-b275-a8a9620f242e" (UID: "1e0f98b9-19d3-4070-b275-a8a9620f242e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:03:48.930779 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:48.930761    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0f98b9-19d3-4070-b275-a8a9620f242e-kube-api-access-kl9p2" pod "1e0f98b9-19d3-4070-b275-a8a9620f242e" (UID: "1e0f98b9-19d3-4070-b275-a8a9620f242e"). InnerVolumeSpecName "kube-api-access-kl9p2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:03:49.029206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:49.029172    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0f98b9-19d3-4070-b275-a8a9620f242e-maas-api-xks-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:49.029206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:49.029207    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kl9p2\" (UniqueName: \"kubernetes.io/projected/1e0f98b9-19d3-4070-b275-a8a9620f242e-kube-api-access-kl9p2\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:03:49.115990 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:03:49.115937    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:03:49.357474 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:49.357444    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"]
Nov 01 13:03:49.366881 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:49.366855    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"]
Nov 01 13:03:49.659811 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:49.659722    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5" event={"ID":"1e0f98b9-19d3-4070-b275-a8a9620f242e","Type":"ContainerDied","Data":"c6af512bd36fa6b347a4ec34eec42c4fd3b79c0f683eb012499d11aaa47d618c"}
Nov 01 13:03:49.659811 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:49.659755    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"
Nov 01 13:03:49.659811 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:03:49.659796    4095 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6af512bd36fa6b347a4ec34eec42c4fd3b79c0f683eb012499d11aaa47d618c"
Nov 01 13:04:04.377568 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.377526    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" containerID="cri-o://6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88" gracePeriod=45
Nov 01 13:04:04.386007 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.385937    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main" containerID="cri-o://5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c" gracePeriod=45
Nov 01 13:04:04.505278 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.505233    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": dial tcp 10.134.0.60:8000: connect: connection refused"
Nov 01 13:04:04.639152 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.639085    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 13:04:04.661164 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.661138    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 13:04:04.744492 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.744458    4095 generic.go:356] "Generic (PLEG): container finished" podID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerID="6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88" exitCode=0
Nov 01 13:04:04.744619 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.744510    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"
Nov 01 13:04:04.744619 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.744547    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" event={"ID":"ce5ce330-ce56-42cb-8e11-38e3a1d0971f","Type":"ContainerDied","Data":"6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88"}
Nov 01 13:04:04.744619 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.744598    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4" event={"ID":"ce5ce330-ce56-42cb-8e11-38e3a1d0971f","Type":"ContainerDied","Data":"fd6d41f9fe928e22406743533b03cd50067e84911ef7605942b6e9e2082cbd0f"}
Nov 01 13:04:04.744619 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.744617    4095 scope.go:122] "RemoveContainer" containerID="6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88"
Nov 01 13:04:04.745852 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.745831    4095 generic.go:356] "Generic (PLEG): container finished" podID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerID="5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c" exitCode=0
Nov 01 13:04:04.746005 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.745863    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" event={"ID":"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7","Type":"ContainerDied","Data":"5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c"}
Nov 01 13:04:04.746005 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.745875    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"
Nov 01 13:04:04.746005 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.745895    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l" event={"ID":"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7","Type":"ContainerDied","Data":"c5b4a62fca6bc25795d56d746db3bdd9693990d104ec22631b4b32202037dc65"}
Nov 01 13:04:04.755078 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.755057    4095 scope.go:122] "RemoveContainer" containerID="6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88"
Nov 01 13:04:04.755417 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:04:04.755399    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88\": container with ID starting with 6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88 not found: ID does not exist" containerID="6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88"
Nov 01 13:04:04.755515 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.755427    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88"} err="failed to get container status \"6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88\": rpc error: code = NotFound desc = could not find container \"6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88\": container with ID starting with 6ab5cb8221db0d16c4b1f4ff2f46393db4d70d278fa12f9b25fc85df28f2de88 not found: ID does not exist"
Nov 01 13:04:04.755515 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.755451    4095 scope.go:122] "RemoveContainer" containerID="5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c"
Nov 01 13:04:04.766469 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.766447    4095 scope.go:122] "RemoveContainer" containerID="5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c"
Nov 01 13:04:04.766773 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:04:04.766753    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c\": container with ID starting with 5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c not found: ID does not exist" containerID="5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c"
Nov 01 13:04:04.766833 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.766784    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c"} err="failed to get container status \"5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c\": rpc error: code = NotFound desc = could not find container \"5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c\": container with ID starting with 5af479ab644b97eb978cee0ebaf949d2f43ea5df8fe24eaf00d6581857a19e2c not found: ID does not exist"
Nov 01 13:04:04.816781 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.816742    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-model-cache\") pod \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") "
Nov 01 13:04:04.816781 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.816784    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tls-certs\") pod \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") "
Nov 01 13:04:04.817029 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.816826    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tls-certs\") pod \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") "
Nov 01 13:04:04.817029 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.816860    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-home\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-home\") pod \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") "
Nov 01 13:04:04.817029 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.816890    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-dshm\") pod \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") "
Nov 01 13:04:04.817029 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.816941    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-kube-api-access-fv5fz\" (UniqueName: \"kubernetes.io/projected/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-kube-api-access-fv5fz\") pod \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") "
Nov 01 13:04:04.817263 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.817086    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-model-cache" pod "ce5ce330-ce56-42cb-8e11-38e3a1d0971f" (UID: "ce5ce330-ce56-42cb-8e11-38e3a1d0971f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.817263 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.817211    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-home" pod "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" (UID: "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.817379 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.817332    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-kube-api-access-2nmbb\" (UniqueName: \"kubernetes.io/projected/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-kube-api-access-2nmbb\") pod \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") "
Nov 01 13:04:04.817441 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.817392    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-model-cache\") pod \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") "
Nov 01 13:04:04.817491 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.817436    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tmp-dir\") pod \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") "
Nov 01 13:04:04.817547 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.817509    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-home\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-home\") pod \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\" (UID: \"ce5ce330-ce56-42cb-8e11-38e3a1d0971f\") "
Nov 01 13:04:04.817602 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.817583    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tmp-dir\") pod \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") "
Nov 01 13:04:04.817661 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.817620    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-dshm\") pod \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\" (UID: \"ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7\") "
Nov 01 13:04:04.818502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.817882    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-model-cache" pod "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" (UID: "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.818502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.818206    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tmp-dir" pod "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" (UID: "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.818502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.818360    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.818502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.818381    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.818502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.818395    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.818502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.818409    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.818502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.818467    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-home" pod "ce5ce330-ce56-42cb-8e11-38e3a1d0971f" (UID: "ce5ce330-ce56-42cb-8e11-38e3a1d0971f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.819240 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.819213    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tmp-dir" pod "ce5ce330-ce56-42cb-8e11-38e3a1d0971f" (UID: "ce5ce330-ce56-42cb-8e11-38e3a1d0971f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.819749 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.819727    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-dshm" pod "ce5ce330-ce56-42cb-8e11-38e3a1d0971f" (UID: "ce5ce330-ce56-42cb-8e11-38e3a1d0971f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.819915 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.819890    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tls-certs" pod "ce5ce330-ce56-42cb-8e11-38e3a1d0971f" (UID: "ce5ce330-ce56-42cb-8e11-38e3a1d0971f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:04:04.820222 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.820198    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-dshm" pod "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" (UID: "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:04:04.820307 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.820238    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tls-certs" pod "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" (UID: "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:04:04.820359 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.820327    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-kube-api-access-fv5fz" pod "ce5ce330-ce56-42cb-8e11-38e3a1d0971f" (UID: "ce5ce330-ce56-42cb-8e11-38e3a1d0971f"). InnerVolumeSpecName "kube-api-access-fv5fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:04:04.820942 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.820912    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-kube-api-access-2nmbb" pod "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" (UID: "ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7"). InnerVolumeSpecName "kube-api-access-2nmbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:04:04.919095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.919017    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.919095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.919046    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.919095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.919071    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv5fz\" (UniqueName: \"kubernetes.io/projected/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-kube-api-access-fv5fz\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.919095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.919082    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2nmbb\" (UniqueName: \"kubernetes.io/projected/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-kube-api-access-2nmbb\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.919095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.919092    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.919095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.919100    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ce5ce330-ce56-42cb-8e11-38e3a1d0971f-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.919394 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.919108    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.919394 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.919116    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:04.953630 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.953592    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"]
Nov 01 13:04:04.954360 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.954343    4095 cpu_manager.go:414] "containerMap: removing container" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main"
Nov 01 13:04:04.954409 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.954363    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main"
Nov 01 13:04:04.954409 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.954398    4095 cpu_manager.go:414] "containerMap: removing container" podUID="1e0f98b9-19d3-4070-b275-a8a9620f242e" containerName="revoke-keys"
Nov 01 13:04:04.954409 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.954406    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0f98b9-19d3-4070-b275-a8a9620f242e" containerName="revoke-keys"
Nov 01 13:04:04.954514 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.954448    4095 cpu_manager.go:414] "containerMap: removing container" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main"
Nov 01 13:04:04.954514 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.954456    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main"
Nov 01 13:04:04.954586 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.954575    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" containerName="main"
Nov 01 13:04:04.954622 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.954593    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" containerName="main"
Nov 01 13:04:04.954622 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.954606    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0f98b9-19d3-4070-b275-a8a9620f242e" containerName="revoke-keys"
Nov 01 13:04:04.959767 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:04.959743    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.012224 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.012185    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"]
Nov 01 13:04:05.014003 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.013951    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:04:05.014183 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.014018    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:04:05.121628 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.121586    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfw7q\" (UniqueName: \"kubernetes.io/projected/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-kube-api-access-sfw7q\") pod \"maas-api-revoke-sub-6af02137-xrjlc\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.121801 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.121651    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af02137-xrjlc\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.121898 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.121798    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af02137-xrjlc\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.180056 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.179960    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"]
Nov 01 13:04:05.184548 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.184505    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/premium-w4-06fd6b-kserve-569c479d6-nr96l"]
Nov 01 13:04:05.223131 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.223099    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfw7q\" (UniqueName: \"kubernetes.io/projected/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-kube-api-access-sfw7q\") pod \"maas-api-revoke-sub-6af02137-xrjlc\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.223288 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.223188    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af02137-xrjlc\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.223288 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.223262    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af02137-xrjlc\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.223918 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.223885    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6af02137-xrjlc\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.225546 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.225528    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6af02137-xrjlc\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.278093 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.278069    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfw7q\" (UniqueName: \"kubernetes.io/projected/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-kube-api-access-sfw7q\") pod \"maas-api-revoke-sub-6af02137-xrjlc\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.288652 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.288626    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"]
Nov 01 13:04:05.290730 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.290709    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-06fd6b/distinct-w4-06fd6b-kserve-7f7459846f-xzml4"]
Nov 01 13:04:05.574818 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.574744    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:05.972919 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:05.972865    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"]
Nov 01 13:04:05.973540 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:04:05.973512    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2fa8148_9ba9_4cfc_a629_7448cf6c5f8a.slice/crio-7815aebae4163e2d32078dd97b7f3989027e701810a9231371988bc7399a9901 WatchSource:0}: Error finding container 7815aebae4163e2d32078dd97b7f3989027e701810a9231371988bc7399a9901: Status 404 returned error can't find the container with id 7815aebae4163e2d32078dd97b7f3989027e701810a9231371988bc7399a9901
Nov 01 13:04:06.188465 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:06.188435    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7" path="/var/lib/kubelet/pods/ba3a9abf-9e99-4e4d-832c-bcdb3c8673e7/volumes"
Nov 01 13:04:06.188850 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:06.188837    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce5ce330-ce56-42cb-8e11-38e3a1d0971f" path="/var/lib/kubelet/pods/ce5ce330-ce56-42cb-8e11-38e3a1d0971f/volumes"
Nov 01 13:04:06.760333 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:06.760298    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc" event={"ID":"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a","Type":"ContainerStarted","Data":"93932b579874742b833d1bb8dbd55e3faff4bb8c4c005989b6e9ebfc703ee89d"}
Nov 01 13:04:06.760754 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:06.760349    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc" event={"ID":"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a","Type":"ContainerStarted","Data":"7815aebae4163e2d32078dd97b7f3989027e701810a9231371988bc7399a9901"}
Nov 01 13:04:06.872109 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:06.872054    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc" podStartSLOduration=2.872036235 podStartE2EDuration="2.872036235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:04:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:04:06.869638997 +0000 UTC m=+2506.787434662" watchObservedRunningTime="2026-11-01 13:04:06.872036235 +0000 UTC m=+2506.789831902"
Nov 01 13:04:07.767717 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:07.767685    4095 generic.go:356] "Generic (PLEG): container finished" podID="f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a" containerID="93932b579874742b833d1bb8dbd55e3faff4bb8c4c005989b6e9ebfc703ee89d" exitCode=0
Nov 01 13:04:07.768160 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:07.767773    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc" event={"ID":"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a","Type":"ContainerDied","Data":"93932b579874742b833d1bb8dbd55e3faff4bb8c4c005989b6e9ebfc703ee89d"}
Nov 01 13:04:08.909466 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:08.909440    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:08.965186 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:08.965156    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-kube-api-access-sfw7q\" (UniqueName: \"kubernetes.io/projected/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-kube-api-access-sfw7q\") pod \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") "
Nov 01 13:04:08.965364 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:08.965221    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-xks-ca\") pod \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") "
Nov 01 13:04:08.965364 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:08.965274    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-openshift-ca\") pod \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\" (UID: \"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a\") "
Nov 01 13:04:08.965719 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:08.965690    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-openshift-ca" pod "f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a" (UID: "f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:04:08.967343 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:08.967317    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-xks-ca" pod "f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a" (UID: "f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:04:08.967460 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:08.967364    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-kube-api-access-sfw7q" pod "f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a" (UID: "f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a"). InnerVolumeSpecName "kube-api-access-sfw7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:04:09.066821 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:09.066744    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfw7q\" (UniqueName: \"kubernetes.io/projected/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-kube-api-access-sfw7q\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:09.066821 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:09.066775    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-xks-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:09.066821 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:09.066785    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a-maas-api-openshift-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:09.779947 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:09.779922    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"
Nov 01 13:04:09.780107 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:09.779915    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc" event={"ID":"f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a","Type":"ContainerDied","Data":"7815aebae4163e2d32078dd97b7f3989027e701810a9231371988bc7399a9901"}
Nov 01 13:04:09.780107 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:09.780044    4095 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7815aebae4163e2d32078dd97b7f3989027e701810a9231371988bc7399a9901"
Nov 01 13:04:19.113410 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:04:19.113368    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:04:21.347712 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.347638    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"]
Nov 01 13:04:21.348173 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.348159    4095 cpu_manager.go:414] "containerMap: removing container" podUID="f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a" containerName="revoke-keys"
Nov 01 13:04:21.348214 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.348173    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a" containerName="revoke-keys"
Nov 01 13:04:21.348266 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.348256    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a" containerName="revoke-keys"
Nov 01 13:04:21.352657 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.352642    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.402416 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.402389    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5b6l8\" (UniqueName: \"kubernetes.io/projected/93f48878-03df-4294-b751-95d3517ea79f-kube-api-access-5b6l8\") pod \"maas-api-revoke-sub-937d9115-g4tsl\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.402576 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.402474    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f48878-03df-4294-b751-95d3517ea79f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937d9115-g4tsl\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.402576 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.402539    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f48878-03df-4294-b751-95d3517ea79f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937d9115-g4tsl\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.402964 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.402946    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:04:21.406559 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.406536    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"]
Nov 01 13:04:21.503882 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.503846    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:04:21.504095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.504075    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5b6l8\" (UniqueName: \"kubernetes.io/projected/93f48878-03df-4294-b751-95d3517ea79f-kube-api-access-5b6l8\") pod \"maas-api-revoke-sub-937d9115-g4tsl\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.504145 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.504106    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f48878-03df-4294-b751-95d3517ea79f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937d9115-g4tsl\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.504145 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.504139    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f48878-03df-4294-b751-95d3517ea79f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937d9115-g4tsl\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.504745 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.504715    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f48878-03df-4294-b751-95d3517ea79f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-937d9115-g4tsl\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.506504 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.506482    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f48878-03df-4294-b751-95d3517ea79f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-937d9115-g4tsl\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.568469 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.568438    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5b6l8\" (UniqueName: \"kubernetes.io/projected/93f48878-03df-4294-b751-95d3517ea79f-kube-api-access-5b6l8\") pod \"maas-api-revoke-sub-937d9115-g4tsl\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:21.665437 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:21.665394    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:22.032196 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:22.032165    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"]
Nov 01 13:04:22.033370 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:04:22.033345    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f48878_03df_4294_b751_95d3517ea79f.slice/crio-d46446d2d14421feb7f06b9056ca6824a1c9f9261f54665e0d74663536a91710 WatchSource:0}: Error finding container d46446d2d14421feb7f06b9056ca6824a1c9f9261f54665e0d74663536a91710: Status 404 returned error can't find the container with id d46446d2d14421feb7f06b9056ca6824a1c9f9261f54665e0d74663536a91710
Nov 01 13:04:22.852200 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:22.852167    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl" event={"ID":"93f48878-03df-4294-b751-95d3517ea79f","Type":"ContainerStarted","Data":"62b12926e72a18e66195385c18c3ff20ff68e97ac008beca7382bf782b16d7ef"}
Nov 01 13:04:22.852588 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:22.852217    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl" event={"ID":"93f48878-03df-4294-b751-95d3517ea79f","Type":"ContainerStarted","Data":"d46446d2d14421feb7f06b9056ca6824a1c9f9261f54665e0d74663536a91710"}
Nov 01 13:04:22.960126 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:22.960083    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl" podStartSLOduration=1.960069839 podStartE2EDuration="1.960069839s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:04:22.959204483 +0000 UTC m=+2522.877000148" watchObservedRunningTime="2026-11-01 13:04:22.960069839 +0000 UTC m=+2522.877865505"
Nov 01 13:04:23.859408 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:23.859373    4095 generic.go:356] "Generic (PLEG): container finished" podID="93f48878-03df-4294-b751-95d3517ea79f" containerID="62b12926e72a18e66195385c18c3ff20ff68e97ac008beca7382bf782b16d7ef" exitCode=0
Nov 01 13:04:23.859812 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:23.859440    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl" event={"ID":"93f48878-03df-4294-b751-95d3517ea79f","Type":"ContainerDied","Data":"62b12926e72a18e66195385c18c3ff20ff68e97ac008beca7382bf782b16d7ef"}
Nov 01 13:04:25.005406 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.005384    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:25.146548 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.146467    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f48878-03df-4294-b751-95d3517ea79f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f48878-03df-4294-b751-95d3517ea79f-maas-api-openshift-ca\") pod \"93f48878-03df-4294-b751-95d3517ea79f\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") "
Nov 01 13:04:25.146720 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.146554    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93f48878-03df-4294-b751-95d3517ea79f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f48878-03df-4294-b751-95d3517ea79f-maas-api-xks-ca\") pod \"93f48878-03df-4294-b751-95d3517ea79f\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") "
Nov 01 13:04:25.146720 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.146600    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f48878-03df-4294-b751-95d3517ea79f-kube-api-access-5b6l8\" (UniqueName: \"kubernetes.io/projected/93f48878-03df-4294-b751-95d3517ea79f-kube-api-access-5b6l8\") pod \"93f48878-03df-4294-b751-95d3517ea79f\" (UID: \"93f48878-03df-4294-b751-95d3517ea79f\") "
Nov 01 13:04:25.146855 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.146816    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f48878-03df-4294-b751-95d3517ea79f-maas-api-openshift-ca" pod "93f48878-03df-4294-b751-95d3517ea79f" (UID: "93f48878-03df-4294-b751-95d3517ea79f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:04:25.146995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.146953    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f48878-03df-4294-b751-95d3517ea79f-maas-api-openshift-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:25.148603 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.148580    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f48878-03df-4294-b751-95d3517ea79f-maas-api-xks-ca" pod "93f48878-03df-4294-b751-95d3517ea79f" (UID: "93f48878-03df-4294-b751-95d3517ea79f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:04:25.148699 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.148646    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f48878-03df-4294-b751-95d3517ea79f-kube-api-access-5b6l8" pod "93f48878-03df-4294-b751-95d3517ea79f" (UID: "93f48878-03df-4294-b751-95d3517ea79f"). InnerVolumeSpecName "kube-api-access-5b6l8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:04:25.248056 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.248031    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5b6l8\" (UniqueName: \"kubernetes.io/projected/93f48878-03df-4294-b751-95d3517ea79f-kube-api-access-5b6l8\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:25.248056 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.248053    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f48878-03df-4294-b751-95d3517ea79f-maas-api-xks-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:25.871503 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.871463    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl" event={"ID":"93f48878-03df-4294-b751-95d3517ea79f","Type":"ContainerDied","Data":"d46446d2d14421feb7f06b9056ca6824a1c9f9261f54665e0d74663536a91710"}
Nov 01 13:04:25.871671 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.871541    4095 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d46446d2d14421feb7f06b9056ca6824a1c9f9261f54665e0d74663536a91710"
Nov 01 13:04:25.871671 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:25.871482    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"
Nov 01 13:04:32.361851 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.361819    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"]
Nov 01 13:04:32.364239 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.362018    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g" podUID="665bb20c-fefd-4732-be3b-553c13774140" containerName="maas-api" containerID="cri-o://7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0" gracePeriod=30
Nov 01 13:04:32.631022 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.630951    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 13:04:32.731140 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.731113    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/665bb20c-fefd-4732-be3b-553c13774140-kube-api-access-tgkk9\" (UniqueName: \"kubernetes.io/projected/665bb20c-fefd-4732-be3b-553c13774140-kube-api-access-tgkk9\") pod \"665bb20c-fefd-4732-be3b-553c13774140\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") "
Nov 01 13:04:32.731310 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.731206    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/665bb20c-fefd-4732-be3b-553c13774140-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665bb20c-fefd-4732-be3b-553c13774140-openshift-service-ca\") pod \"665bb20c-fefd-4732-be3b-553c13774140\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") "
Nov 01 13:04:32.731310 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.731235    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-metrics-certs\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-metrics-certs\") pod \"665bb20c-fefd-4732-be3b-553c13774140\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") "
Nov 01 13:04:32.731310 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.731279    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-maas-api-tls\") pod \"665bb20c-fefd-4732-be3b-553c13774140\" (UID: \"665bb20c-fefd-4732-be3b-553c13774140\") "
Nov 01 13:04:32.731622 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.731592    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/665bb20c-fefd-4732-be3b-553c13774140-openshift-service-ca" pod "665bb20c-fefd-4732-be3b-553c13774140" (UID: "665bb20c-fefd-4732-be3b-553c13774140"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:04:32.731777 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.731744    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/665bb20c-fefd-4732-be3b-553c13774140-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:32.733356 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.733327    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-maas-api-tls" pod "665bb20c-fefd-4732-be3b-553c13774140" (UID: "665bb20c-fefd-4732-be3b-553c13774140"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:04:32.733458 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.733351    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/665bb20c-fefd-4732-be3b-553c13774140-kube-api-access-tgkk9" pod "665bb20c-fefd-4732-be3b-553c13774140" (UID: "665bb20c-fefd-4732-be3b-553c13774140"). InnerVolumeSpecName "kube-api-access-tgkk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:04:32.733458 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.733413    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-metrics-certs" pod "665bb20c-fefd-4732-be3b-553c13774140" (UID: "665bb20c-fefd-4732-be3b-553c13774140"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:04:32.833188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.833159    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:32.833188 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.833190    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgkk9\" (UniqueName: \"kubernetes.io/projected/665bb20c-fefd-4732-be3b-553c13774140-kube-api-access-tgkk9\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:32.833375 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.833205    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/665bb20c-fefd-4732-be3b-553c13774140-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:04:32.917886 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.917751    4095 generic.go:356] "Generic (PLEG): container finished" podID="665bb20c-fefd-4732-be3b-553c13774140" containerID="7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0" exitCode=0
Nov 01 13:04:32.917886 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.917811    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"
Nov 01 13:04:32.917886 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.917845    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g" event={"ID":"665bb20c-fefd-4732-be3b-553c13774140","Type":"ContainerDied","Data":"7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0"}
Nov 01 13:04:32.919647 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.917932    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g" event={"ID":"665bb20c-fefd-4732-be3b-553c13774140","Type":"ContainerDied","Data":"6598fdcb18d1b637cc65b860b517de7fd768df1f14a249e1b22499270efe4d5c"}
Nov 01 13:04:32.919647 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.917961    4095 scope.go:122] "RemoveContainer" containerID="7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0"
Nov 01 13:04:32.929287 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.929270    4095 scope.go:122] "RemoveContainer" containerID="7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0"
Nov 01 13:04:32.929616 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:04:32.929595    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0\": container with ID starting with 7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0 not found: ID does not exist" containerID="7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0"
Nov 01 13:04:32.929699 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:32.929623    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0"} err="failed to get container status \"7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0\": rpc error: code = NotFound desc = could not find container \"7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0\": container with ID starting with 7ed9c8054e07f85ba0ca055f1b62ac40e87b28ec7174cc44f41c136954e517a0 not found: ID does not exist"
Nov 01 13:04:33.074116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:33.074084    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"]
Nov 01 13:04:33.077614 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:33.077588    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-ecd640-787c85f684-h6q4g"]
Nov 01 13:04:34.188949 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:34.188917    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="665bb20c-fefd-4732-be3b-553c13774140" path="/var/lib/kubelet/pods/665bb20c-fefd-4732-be3b-553c13774140/volumes"
Nov 01 13:04:34.890172 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:34.890109    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"]
Nov 01 13:04:34.937357 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:34.937327    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-ecd640-6752g"]
Nov 01 13:04:36.195508 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:36.195473    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db4916da-46d6-4c6f-b970-0f504b3efb56" path="/var/lib/kubelet/pods/db4916da-46d6-4c6f-b970-0f504b3efb56/volumes"
Nov 01 13:04:49.114905 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:04:49.114862    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:04:52.353470 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:52.353434    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"]
Nov 01 13:04:52.400626 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:52.400594    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-06fd6b-h2wz5"]
Nov 01 13:04:54.196254 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:54.196220    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0f98b9-19d3-4070-b275-a8a9620f242e" path="/var/lib/kubelet/pods/1e0f98b9-19d3-4070-b275-a8a9620f242e/volumes"
Nov 01 13:04:58.448680 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.448636    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"]
Nov 01 13:04:58.449413 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.449383    4095 cpu_manager.go:414] "containerMap: removing container" podUID="93f48878-03df-4294-b751-95d3517ea79f" containerName="revoke-keys"
Nov 01 13:04:58.449413 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.449403    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f48878-03df-4294-b751-95d3517ea79f" containerName="revoke-keys"
Nov 01 13:04:58.449624 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.449455    4095 cpu_manager.go:414] "containerMap: removing container" podUID="665bb20c-fefd-4732-be3b-553c13774140" containerName="maas-api"
Nov 01 13:04:58.449624 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.449463    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="665bb20c-fefd-4732-be3b-553c13774140" containerName="maas-api"
Nov 01 13:04:58.449771 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.449753    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="665bb20c-fefd-4732-be3b-553c13774140" containerName="maas-api"
Nov 01 13:04:58.449837 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.449796    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f48878-03df-4294-b751-95d3517ea79f" containerName="revoke-keys"
Nov 01 13:04:58.453812 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.453790    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.505731 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.505688    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"]
Nov 01 13:04:58.550578 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.550545    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:04:58.550792 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.550774    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:04:58.598328 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.598289    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nccdg\" (UniqueName: \"kubernetes.io/projected/bf3d8e04-4e53-4345-b458-a61f71a67f4f-kube-api-access-nccdg\") pod \"maas-api-revoke-sub-fc7f81ce-wlg9d\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.598496 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.598348    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc7f81ce-wlg9d\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.598496 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.598449    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc7f81ce-wlg9d\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.699959 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.699882    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nccdg\" (UniqueName: \"kubernetes.io/projected/bf3d8e04-4e53-4345-b458-a61f71a67f4f-kube-api-access-nccdg\") pod \"maas-api-revoke-sub-fc7f81ce-wlg9d\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.699959 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.699941    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc7f81ce-wlg9d\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.700160 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.700002    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc7f81ce-wlg9d\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.700721 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.700700    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fc7f81ce-wlg9d\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.702396 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.702374    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fc7f81ce-wlg9d\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.754749 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.754713    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nccdg\" (UniqueName: \"kubernetes.io/projected/bf3d8e04-4e53-4345-b458-a61f71a67f4f-kube-api-access-nccdg\") pod \"maas-api-revoke-sub-fc7f81ce-wlg9d\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:58.767517 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:58.767487    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:04:59.133018 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:04:59.132990    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"]
Nov 01 13:04:59.134403 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:04:59.134375    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf3d8e04_4e53_4345_b458_a61f71a67f4f.slice/crio-f005d762b21c137bb8e85e9e594f515d0b821b94bd82599916011eb4b4c3ede7 WatchSource:0}: Error finding container f005d762b21c137bb8e85e9e594f515d0b821b94bd82599916011eb4b4c3ede7: Status 404 returned error can't find the container with id f005d762b21c137bb8e85e9e594f515d0b821b94bd82599916011eb4b4c3ede7
Nov 01 13:05:00.075428 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:00.075387    4095 generic.go:356] "Generic (PLEG): container finished" podID="bf3d8e04-4e53-4345-b458-a61f71a67f4f" containerID="226fbb954068ccb638b048922ccf220c52edab8fb01e8b804790c311e7f15c79" exitCode=0
Nov 01 13:05:00.075857 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:00.075467    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d" event={"ID":"bf3d8e04-4e53-4345-b458-a61f71a67f4f","Type":"ContainerDied","Data":"226fbb954068ccb638b048922ccf220c52edab8fb01e8b804790c311e7f15c79"}
Nov 01 13:05:00.075857 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:00.075524    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d" event={"ID":"bf3d8e04-4e53-4345-b458-a61f71a67f4f","Type":"ContainerStarted","Data":"f005d762b21c137bb8e85e9e594f515d0b821b94bd82599916011eb4b4c3ede7"}
Nov 01 13:05:01.215063 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.215036    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:05:01.340666 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.340592    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-openshift-ca\") pod \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") "
Nov 01 13:05:01.340666 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.340629    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-xks-ca\") pod \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") "
Nov 01 13:05:01.340892 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.340743    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf3d8e04-4e53-4345-b458-a61f71a67f4f-kube-api-access-nccdg\" (UniqueName: \"kubernetes.io/projected/bf3d8e04-4e53-4345-b458-a61f71a67f4f-kube-api-access-nccdg\") pod \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\" (UID: \"bf3d8e04-4e53-4345-b458-a61f71a67f4f\") "
Nov 01 13:05:01.341011 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.340958    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-openshift-ca" pod "bf3d8e04-4e53-4345-b458-a61f71a67f4f" (UID: "bf3d8e04-4e53-4345-b458-a61f71a67f4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:05:01.341378 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.341348    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-openshift-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:01.343161 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.343139    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-xks-ca" pod "bf3d8e04-4e53-4345-b458-a61f71a67f4f" (UID: "bf3d8e04-4e53-4345-b458-a61f71a67f4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:05:01.343285 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.343272    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf3d8e04-4e53-4345-b458-a61f71a67f4f-kube-api-access-nccdg" pod "bf3d8e04-4e53-4345-b458-a61f71a67f4f" (UID: "bf3d8e04-4e53-4345-b458-a61f71a67f4f"). InnerVolumeSpecName "kube-api-access-nccdg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:05:01.442348 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.442320    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf3d8e04-4e53-4345-b458-a61f71a67f4f-maas-api-xks-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:01.442348 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:01.442343    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nccdg\" (UniqueName: \"kubernetes.io/projected/bf3d8e04-4e53-4345-b458-a61f71a67f4f-kube-api-access-nccdg\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:02.086830 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:02.086798    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"
Nov 01 13:05:02.086830 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:02.086818    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d" event={"ID":"bf3d8e04-4e53-4345-b458-a61f71a67f4f","Type":"ContainerDied","Data":"f005d762b21c137bb8e85e9e594f515d0b821b94bd82599916011eb4b4c3ede7"}
Nov 01 13:05:02.087084 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:02.086865    4095 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f005d762b21c137bb8e85e9e594f515d0b821b94bd82599916011eb4b4c3ede7"
Nov 01 13:05:11.503087 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.503051    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"]
Nov 01 13:05:11.504314 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.504287    4095 cpu_manager.go:414] "containerMap: removing container" podUID="bf3d8e04-4e53-4345-b458-a61f71a67f4f" containerName="revoke-keys"
Nov 01 13:05:11.504460 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.504447    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf3d8e04-4e53-4345-b458-a61f71a67f4f" containerName="revoke-keys"
Nov 01 13:05:11.504682 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.504671    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf3d8e04-4e53-4345-b458-a61f71a67f4f" containerName="revoke-keys"
Nov 01 13:05:11.508352 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.508332    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.552165 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.552136    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b3222c-vb95q\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.552282 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.552170    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mwcb\" (UniqueName: \"kubernetes.io/projected/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-kube-api-access-4mwcb\") pod \"maas-api-revoke-sub-a5b3222c-vb95q\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.552282 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.552210    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b3222c-vb95q\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.606032 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.605963    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:05:11.606228 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.606066    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:05:11.608470 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.608450    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"]
Nov 01 13:05:11.653548 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.653512    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b3222c-vb95q\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.653548 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.653548    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mwcb\" (UniqueName: \"kubernetes.io/projected/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-kube-api-access-4mwcb\") pod \"maas-api-revoke-sub-a5b3222c-vb95q\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.653817 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.653585    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b3222c-vb95q\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.654268 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.654239    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a5b3222c-vb95q\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.655903 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.655883    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a5b3222c-vb95q\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.708772 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.708736    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mwcb\" (UniqueName: \"kubernetes.io/projected/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-kube-api-access-4mwcb\") pod \"maas-api-revoke-sub-a5b3222c-vb95q\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:11.823075 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:11.822991    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:12.187693 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:05:12.187666    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b31eaa1_9402_4eef_a81e_9e8dfcff14b7.slice/crio-3683558f1b268d0cf4ddffa3088aa8e97b329a100c9966c4c52500cac8079e82 WatchSource:0}: Error finding container 3683558f1b268d0cf4ddffa3088aa8e97b329a100c9966c4c52500cac8079e82: Status 404 returned error can't find the container with id 3683558f1b268d0cf4ddffa3088aa8e97b329a100c9966c4c52500cac8079e82
Nov 01 13:05:12.189742 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:12.189706    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"]
Nov 01 13:05:13.150914 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:13.150876    4095 generic.go:356] "Generic (PLEG): container finished" podID="0b31eaa1-9402-4eef-a81e-9e8dfcff14b7" containerID="ee6f056f4c7579a51d570166e782f180c46da61e7cb218cd7c991b04d504672b" exitCode=0
Nov 01 13:05:13.151346 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:13.150961    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q" event={"ID":"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7","Type":"ContainerDied","Data":"ee6f056f4c7579a51d570166e782f180c46da61e7cb218cd7c991b04d504672b"}
Nov 01 13:05:13.151346 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:13.151024    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q" event={"ID":"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7","Type":"ContainerStarted","Data":"3683558f1b268d0cf4ddffa3088aa8e97b329a100c9966c4c52500cac8079e82"}
Nov 01 13:05:14.294763 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.294738    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:14.385895 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.385859    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-kube-api-access-4mwcb\" (UniqueName: \"kubernetes.io/projected/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-kube-api-access-4mwcb\") pod \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") "
Nov 01 13:05:14.386113 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.385924    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-xks-ca\") pod \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") "
Nov 01 13:05:14.386113 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.386010    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-openshift-ca\") pod \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\" (UID: \"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7\") "
Nov 01 13:05:14.386349 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.386326    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-openshift-ca" pod "0b31eaa1-9402-4eef-a81e-9e8dfcff14b7" (UID: "0b31eaa1-9402-4eef-a81e-9e8dfcff14b7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:05:14.388022 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.387999    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-xks-ca" pod "0b31eaa1-9402-4eef-a81e-9e8dfcff14b7" (UID: "0b31eaa1-9402-4eef-a81e-9e8dfcff14b7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:05:14.388095 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.388049    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-kube-api-access-4mwcb" pod "0b31eaa1-9402-4eef-a81e-9e8dfcff14b7" (UID: "0b31eaa1-9402-4eef-a81e-9e8dfcff14b7"). InnerVolumeSpecName "kube-api-access-4mwcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:05:14.487222 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.487137    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mwcb\" (UniqueName: \"kubernetes.io/projected/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-kube-api-access-4mwcb\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:14.487222 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.487170    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-xks-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:14.487222 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:14.487186    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7-maas-api-openshift-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:15.165061 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:15.165034    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"
Nov 01 13:05:15.165235 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:15.165027    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q" event={"ID":"0b31eaa1-9402-4eef-a81e-9e8dfcff14b7","Type":"ContainerDied","Data":"3683558f1b268d0cf4ddffa3088aa8e97b329a100c9966c4c52500cac8079e82"}
Nov 01 13:05:15.165235 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:15.165155    4095 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3683558f1b268d0cf4ddffa3088aa8e97b329a100c9966c4c52500cac8079e82"
Nov 01 13:05:19.113562 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:05:19.113526    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:05:43.069090 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:43.069055    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"]
Nov 01 13:05:43.069598 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:43.069283    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy" containerID="cri-o://220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a" gracePeriod=30
Nov 01 13:05:48.326207 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.326178    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 13:05:48.347784 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.347711    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-certs\") pod \"0d753619-fbf8-4087-85ca-e2a61d971e8d\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") "
Nov 01 13:05:48.347931 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.347781    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-envoy\") pod \"0d753619-fbf8-4087-85ca-e2a61d971e8d\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") "
Nov 01 13:05:48.347931 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.347852    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-token\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-token\") pod \"0d753619-fbf8-4087-85ca-e2a61d971e8d\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") "
Nov 01 13:05:48.347931 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.347893    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-credential-socket\") pod \"0d753619-fbf8-4087-85ca-e2a61d971e8d\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") "
Nov 01 13:05:48.348120 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.347945    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0d753619-fbf8-4087-85ca-e2a61d971e8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d753619-fbf8-4087-85ca-e2a61d971e8d-istiod-ca-cert\") pod \"0d753619-fbf8-4087-85ca-e2a61d971e8d\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") "
Nov 01 13:05:48.348120 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348002    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-certs" pod "0d753619-fbf8-4087-85ca-e2a61d971e8d" (UID: "0d753619-fbf8-4087-85ca-e2a61d971e8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:48.348120 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348079    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-data\") pod \"0d753619-fbf8-4087-85ca-e2a61d971e8d\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") "
Nov 01 13:05:48.348120 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348108    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-kube-api-access-b8rgt\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-kube-api-access-b8rgt\") pod \"0d753619-fbf8-4087-85ca-e2a61d971e8d\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") "
Nov 01 13:05:48.348319 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348154    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-socket\") pod \"0d753619-fbf8-4087-85ca-e2a61d971e8d\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") "
Nov 01 13:05:48.348319 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348228    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-podinfo\") pod \"0d753619-fbf8-4087-85ca-e2a61d971e8d\" (UID: \"0d753619-fbf8-4087-85ca-e2a61d971e8d\") "
Nov 01 13:05:48.348319 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348247    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-credential-socket" pod "0d753619-fbf8-4087-85ca-e2a61d971e8d" (UID: "0d753619-fbf8-4087-85ca-e2a61d971e8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:48.348484 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348394    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-data" pod "0d753619-fbf8-4087-85ca-e2a61d971e8d" (UID: "0d753619-fbf8-4087-85ca-e2a61d971e8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:48.348551 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348527    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-socket" pod "0d753619-fbf8-4087-85ca-e2a61d971e8d" (UID: "0d753619-fbf8-4087-85ca-e2a61d971e8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:48.348704 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348681    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:48.348787 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348711    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:48.348787 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348726    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:48.348787 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348740    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:48.348787 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.348772    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0d753619-fbf8-4087-85ca-e2a61d971e8d-istiod-ca-cert" pod "0d753619-fbf8-4087-85ca-e2a61d971e8d" (UID: "0d753619-fbf8-4087-85ca-e2a61d971e8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:05:48.350185 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.350158    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-envoy" pod "0d753619-fbf8-4087-85ca-e2a61d971e8d" (UID: "0d753619-fbf8-4087-85ca-e2a61d971e8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:05:48.350376 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.350357    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-kube-api-access-b8rgt" pod "0d753619-fbf8-4087-85ca-e2a61d971e8d" (UID: "0d753619-fbf8-4087-85ca-e2a61d971e8d"). InnerVolumeSpecName "kube-api-access-b8rgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:05:48.350531 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.350508    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-token" pod "0d753619-fbf8-4087-85ca-e2a61d971e8d" (UID: "0d753619-fbf8-4087-85ca-e2a61d971e8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:05:48.350710 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.350685    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-podinfo" pod "0d753619-fbf8-4087-85ca-e2a61d971e8d" (UID: "0d753619-fbf8-4087-85ca-e2a61d971e8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:05:48.353025 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.352946    4095 generic.go:356] "Generic (PLEG): container finished" podID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerID="220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a" exitCode=0
Nov 01 13:05:48.353147 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.353014    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"
Nov 01 13:05:48.353223 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.353015    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" event={"ID":"0d753619-fbf8-4087-85ca-e2a61d971e8d","Type":"ContainerDied","Data":"220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a"}
Nov 01 13:05:48.353267 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.353254    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr" event={"ID":"0d753619-fbf8-4087-85ca-e2a61d971e8d","Type":"ContainerDied","Data":"e6eccd67bc8863ab61226db74f98fbae8e8c54c430c46ab805c00e3ff5b5b613"}
Nov 01 13:05:48.353302 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.353280    4095 scope.go:122] "RemoveContainer" containerID="220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a"
Nov 01 13:05:48.372443 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.372422    4095 scope.go:122] "RemoveContainer" containerID="220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a"
Nov 01 13:05:48.372731 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:05:48.372710    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a\": container with ID starting with 220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a not found: ID does not exist" containerID="220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a"
Nov 01 13:05:48.372813 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.372738    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a"} err="failed to get container status \"220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a\": rpc error: code = NotFound desc = could not find container \"220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a\": container with ID starting with 220d09aa09ed037dfdf441678ce9f2f71ee8ef80a4f83253326513dbf946010a not found: ID does not exist"
Nov 01 13:05:48.449306 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.449283    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0d753619-fbf8-4087-85ca-e2a61d971e8d-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:48.449422 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.449308    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8rgt\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-kube-api-access-b8rgt\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:48.449422 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.449324    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:48.449422 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.449336    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:48.449422 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.449347    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0d753619-fbf8-4087-85ca-e2a61d971e8d-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:48.637237 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.637207    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"]
Nov 01 13:05:48.642547 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:48.642523    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7427d-openshift-default-cdb8dd597-7xnrr"]
Nov 01 13:05:49.113399 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:05:49.113367    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:05:49.314547 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:49.314505    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"]
Nov 01 13:05:50.189020 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:50.188990    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" path="/var/lib/kubelet/pods/0d753619-fbf8-4087-85ca-e2a61d971e8d/volumes"
Nov 01 13:05:54.315691 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:54.315613    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm" podUID="004f8295-ef7a-4c55-b1fd-317c99992416" containerName="payload-processing" containerID="cri-o://c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69" gracePeriod=25
Nov 01 13:05:54.572743 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:54.572681    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 13:05:54.619070 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:54.619033    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/004f8295-ef7a-4c55-b1fd-317c99992416-kube-api-access-hmxtw\" (UniqueName: \"kubernetes.io/projected/004f8295-ef7a-4c55-b1fd-317c99992416-kube-api-access-hmxtw\") pod \"004f8295-ef7a-4c55-b1fd-317c99992416\" (UID: \"004f8295-ef7a-4c55-b1fd-317c99992416\") "
Nov 01 13:05:54.619204 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:54.619095    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/004f8295-ef7a-4c55-b1fd-317c99992416-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004f8295-ef7a-4c55-b1fd-317c99992416-plugins-config-volume\") pod \"004f8295-ef7a-4c55-b1fd-317c99992416\" (UID: \"004f8295-ef7a-4c55-b1fd-317c99992416\") "
Nov 01 13:05:54.619488 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:54.619466    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004f8295-ef7a-4c55-b1fd-317c99992416-plugins-config-volume" pod "004f8295-ef7a-4c55-b1fd-317c99992416" (UID: "004f8295-ef7a-4c55-b1fd-317c99992416"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:05:54.621181 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:54.621157    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/004f8295-ef7a-4c55-b1fd-317c99992416-kube-api-access-hmxtw" pod "004f8295-ef7a-4c55-b1fd-317c99992416" (UID: "004f8295-ef7a-4c55-b1fd-317c99992416"). InnerVolumeSpecName "kube-api-access-hmxtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:05:54.720584 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:54.720552    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmxtw\" (UniqueName: \"kubernetes.io/projected/004f8295-ef7a-4c55-b1fd-317c99992416-kube-api-access-hmxtw\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:54.720584 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:54.720579    4095 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/004f8295-ef7a-4c55-b1fd-317c99992416-plugins-config-volume\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:05:55.391777 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:55.391741    4095 generic.go:356] "Generic (PLEG): container finished" podID="004f8295-ef7a-4c55-b1fd-317c99992416" containerID="c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69" exitCode=0
Nov 01 13:05:55.392278 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:55.391787    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"
Nov 01 13:05:55.392278 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:55.391832    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm" event={"ID":"004f8295-ef7a-4c55-b1fd-317c99992416","Type":"ContainerDied","Data":"c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69"}
Nov 01 13:05:55.392278 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:55.391893    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm" event={"ID":"004f8295-ef7a-4c55-b1fd-317c99992416","Type":"ContainerDied","Data":"a8433989a72f3238732d9e4f3eaa37173be2afb30600fe6dfe1880fb95d395d1"}
Nov 01 13:05:55.392278 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:55.391927    4095 scope.go:122] "RemoveContainer" containerID="c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69"
Nov 01 13:05:55.401189 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:55.401169    4095 scope.go:122] "RemoveContainer" containerID="c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69"
Nov 01 13:05:55.401452 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:05:55.401434    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69\": container with ID starting with c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69 not found: ID does not exist" containerID="c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69"
Nov 01 13:05:55.401502 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:55.401466    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69"} err="failed to get container status \"c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69\": rpc error: code = NotFound desc = could not find container \"c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69\": container with ID starting with c9e6b2248f8024c72a2bd5fe2eafff5c3ae566e8fb2181cf4b3874a631e91d69 not found: ID does not exist"
Nov 01 13:05:55.548425 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:55.548394    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"]
Nov 01 13:05:55.595909 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:55.595878    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-72078a-5d6798d6cf-r59sm"]
Nov 01 13:05:56.188419 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:05:56.188380    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="004f8295-ef7a-4c55-b1fd-317c99992416" path="/var/lib/kubelet/pods/004f8295-ef7a-4c55-b1fd-317c99992416/volumes"
Nov 01 13:06:05.719091 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:05.719058    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"]
Nov 01 13:06:05.719862 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:05.719258    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy" containerID="cri-o://de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4" gracePeriod=30
Nov 01 13:06:06.666960 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.666918    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"]
Nov 01 13:06:06.667475 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.667455    4095 cpu_manager.go:414] "containerMap: removing container" podUID="004f8295-ef7a-4c55-b1fd-317c99992416" containerName="payload-processing"
Nov 01 13:06:06.667587 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.667476    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="004f8295-ef7a-4c55-b1fd-317c99992416" containerName="payload-processing"
Nov 01 13:06:06.667587 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.667499    4095 cpu_manager.go:414] "containerMap: removing container" podUID="0b31eaa1-9402-4eef-a81e-9e8dfcff14b7" containerName="revoke-keys"
Nov 01 13:06:06.667587 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.667507    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b31eaa1-9402-4eef-a81e-9e8dfcff14b7" containerName="revoke-keys"
Nov 01 13:06:06.667587 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.667557    4095 cpu_manager.go:414] "containerMap: removing container" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy"
Nov 01 13:06:06.667587 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.667564    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy"
Nov 01 13:06:06.667821 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.667685    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="004f8295-ef7a-4c55-b1fd-317c99992416" containerName="payload-processing"
Nov 01 13:06:06.667821 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.667699    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b31eaa1-9402-4eef-a81e-9e8dfcff14b7" containerName="revoke-keys"
Nov 01 13:06:06.667821 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.667717    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d753619-fbf8-4087-85ca-e2a61d971e8d" containerName="istio-proxy"
Nov 01 13:06:06.672643 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.672620    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.722784 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.722748    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 01 13:06:06.723299 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.723015    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-s5fwc\""
Nov 01 13:06:06.725303 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.725276    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"]
Nov 01 13:06:06.762743 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.762708    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.762928 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.762749    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9rbj\" (UniqueName: \"kubernetes.io/projected/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-kube-api-access-w9rbj\") pod \"maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.763025 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.762933    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.864355 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.864317    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.864538 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.864398    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.864538 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.864423    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9rbj\" (UniqueName: \"kubernetes.io/projected/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-kube-api-access-w9rbj\") pod \"maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.865161 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.865138    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.866752 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.866732    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.918998 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.918929    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9rbj\" (UniqueName: \"kubernetes.io/projected/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-kube-api-access-w9rbj\") pod \"maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:06.985767 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:06.985734    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:07.353713 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:07.353685    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"]
Nov 01 13:06:07.465920 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:07.465882    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb" event={"ID":"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5","Type":"ContainerStarted","Data":"2b8fbb54d19edc750100dbfc4e897b5e4c374c5eb4df0910a789c8bb4e2d2923"}
Nov 01 13:06:08.472925 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:08.472839    4095 generic.go:356] "Generic (PLEG): container finished" podID="5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5" containerID="c7236884dd6688879cc7afe7cdc101a28e81d0c5538e1778715989e8845c7586" exitCode=0
Nov 01 13:06:08.473307 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:08.472922    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb" event={"ID":"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5","Type":"ContainerDied","Data":"c7236884dd6688879cc7afe7cdc101a28e81d0c5538e1778715989e8845c7586"}
Nov 01 13:06:09.612218 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.612192    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:09.796963 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.796864    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-kube-api-access-w9rbj\" (UniqueName: \"kubernetes.io/projected/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-kube-api-access-w9rbj\") pod \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") "
Nov 01 13:06:09.796963 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.796951    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-xks-ca\") pod \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") "
Nov 01 13:06:09.797230 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.797038    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-openshift-ca\") pod \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\" (UID: \"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5\") "
Nov 01 13:06:09.797465 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.797431    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-openshift-ca" pod "5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5" (UID: "5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:06:09.799184 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.799162    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-kube-api-access-w9rbj" pod "5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5" (UID: "5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5"). InnerVolumeSpecName "kube-api-access-w9rbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:06:09.799296 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.799270    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-xks-ca" pod "5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5" (UID: "5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:06:09.899088 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.899048    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9rbj\" (UniqueName: \"kubernetes.io/projected/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-kube-api-access-w9rbj\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:09.899088 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.899088    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-xks-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:09.899335 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:09.899103    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5-maas-api-openshift-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:10.485875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:10.485791    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb" event={"ID":"5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5","Type":"ContainerDied","Data":"2b8fbb54d19edc750100dbfc4e897b5e4c374c5eb4df0910a789c8bb4e2d2923"}
Nov 01 13:06:10.485875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:10.485828    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"
Nov 01 13:06:10.485875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:10.485841    4095 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2b8fbb54d19edc750100dbfc4e897b5e4c374c5eb4df0910a789c8bb4e2d2923"
Nov 01 13:06:10.973270 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:10.973245    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 13:06:11.113083 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113056    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-socket\") pod \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") "
Nov 01 13:06:11.113253 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113092    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-kube-api-access-dxkwg\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-kube-api-access-dxkwg\") pod \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") "
Nov 01 13:06:11.113253 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113125    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-certs\") pod \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") "
Nov 01 13:06:11.113253 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113232    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-podinfo\") pod \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") "
Nov 01 13:06:11.113424 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113262    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istiod-ca-cert\") pod \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") "
Nov 01 13:06:11.113424 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113295    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-data\") pod \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") "
Nov 01 13:06:11.113424 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113337    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-envoy\") pod \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") "
Nov 01 13:06:11.113424 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113373    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-token\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-token\") pod \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") "
Nov 01 13:06:11.113424 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113375    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-socket" pod "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" (UID: "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:06:11.113699 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113424    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-credential-socket\") pod \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\" (UID: \"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6\") "
Nov 01 13:06:11.113699 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113473    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-certs" pod "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" (UID: "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:06:11.113808 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113729    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-data" pod "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" (UID: "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:06:11.113870 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113849    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:11.113933 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113869    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:11.113933 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.113883    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:11.114057 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.114004    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istiod-ca-cert" pod "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" (UID: "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:06:11.114118 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.114067    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-credential-socket" pod "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" (UID: "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:06:11.115371 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.115347    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-kube-api-access-dxkwg" pod "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" (UID: "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6"). InnerVolumeSpecName "kube-api-access-dxkwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:06:11.115492 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.115476    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-podinfo" pod "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" (UID: "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:06:11.115883 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.115864    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-envoy" pod "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" (UID: "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:06:11.115938 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.115892    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-token" pod "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" (UID: "cec0037a-c467-4cb4-bb5b-d556ff5e7cf6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:06:11.215064 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.215033    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxkwg\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-kube-api-access-dxkwg\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:11.215064 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.215059    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:11.215064 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.215070    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:11.215304 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.215079    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:11.215304 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.215088    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:11.215304 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.215096    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:06:11.493726 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.493646    4095 generic.go:356] "Generic (PLEG): container finished" podID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerID="de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4" exitCode=0
Nov 01 13:06:11.493726 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.493698    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"
Nov 01 13:06:11.493914 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.493726    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" event={"ID":"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6","Type":"ContainerDied","Data":"de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4"}
Nov 01 13:06:11.493914 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.493776    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m" event={"ID":"cec0037a-c467-4cb4-bb5b-d556ff5e7cf6","Type":"ContainerDied","Data":"16d7bca712f0c405aa43fc7dff94e35737f6b1c96dfa0e277705abfee8c89de9"}
Nov 01 13:06:11.493914 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.493796    4095 scope.go:122] "RemoveContainer" containerID="de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4"
Nov 01 13:06:11.503789 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.503770    4095 scope.go:122] "RemoveContainer" containerID="d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52"
Nov 01 13:06:11.512032 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.512008    4095 scope.go:122] "RemoveContainer" containerID="de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4"
Nov 01 13:06:11.512297 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:06:11.512275    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4\": container with ID starting with de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4 not found: ID does not exist" containerID="de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4"
Nov 01 13:06:11.512356 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.512305    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4"} err="failed to get container status \"de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4\": rpc error: code = NotFound desc = could not find container \"de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4\": container with ID starting with de450fd2803dde3c0741f09f8710225cc4fd77b89be6400de4368bceacaf0db4 not found: ID does not exist"
Nov 01 13:06:11.512356 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.512322    4095 scope.go:122] "RemoveContainer" containerID="d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52"
Nov 01 13:06:11.512569 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:06:11.512555    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52\": container with ID starting with d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52 not found: ID does not exist" containerID="d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52"
Nov 01 13:06:11.512617 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.512575    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52"} err="failed to get container status \"d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52\": rpc error: code = NotFound desc = could not find container \"d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52\": container with ID starting with d3acd24a517ededf929d5913afd29d2cb4476f1475ee9742a84743ad9711af52 not found: ID does not exist"
Nov 01 13:06:11.657915 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.653075    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"]
Nov 01 13:06:11.698684 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:11.698640    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-72078a-openshift-default-7b9b8d69d9-xjb5m"]
Nov 01 13:06:12.188490 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:12.188455    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" path="/var/lib/kubelet/pods/cec0037a-c467-4cb4-bb5b-d556ff5e7cf6/volumes"
Nov 01 13:06:19.114181 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:06:19.114140    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:06:33.044791 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:33.044716    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"]
Nov 01 13:06:33.091483 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:33.091452    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41d9af8c-s2qzb"]
Nov 01 13:06:33.106911 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:33.106877    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"]
Nov 01 13:06:33.109307 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:33.109270    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b68322-4cbhb"]
Nov 01 13:06:34.188616 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:34.188587    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fdbf1d4-410e-4731-a492-3c425e5090e9" path="/var/lib/kubelet/pods/2fdbf1d4-410e-4731-a492-3c425e5090e9/volumes"
Nov 01 13:06:34.189084 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:34.189068    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5" path="/var/lib/kubelet/pods/5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5/volumes"
Nov 01 13:06:45.211658 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:06:45.211618    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"]
Nov 01 13:06:49.112520 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:06:49.112484    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:07:00.231819 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.231758    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" containerID="cri-o://a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e" gracePeriod=45
Nov 01 13:07:00.256414 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.256385    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused"
Nov 01 13:07:00.482824 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.482774    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 13:07:00.531172 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531145    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-dshm\") pod \"c552b5a6-2b9c-4584-9e8a-d709799b5730\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") "
Nov 01 13:07:00.531294 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531210    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-model-cache\") pod \"c552b5a6-2b9c-4584-9e8a-d709799b5730\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") "
Nov 01 13:07:00.531338 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531327    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c552b5a6-2b9c-4584-9e8a-d709799b5730-kube-api-access-qmrw8\" (UniqueName: \"kubernetes.io/projected/c552b5a6-2b9c-4584-9e8a-d709799b5730-kube-api-access-qmrw8\") pod \"c552b5a6-2b9c-4584-9e8a-d709799b5730\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") "
Nov 01 13:07:00.531383 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531369    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-home\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-home\") pod \"c552b5a6-2b9c-4584-9e8a-d709799b5730\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") "
Nov 01 13:07:00.531432 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531417    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-tmp-dir\") pod \"c552b5a6-2b9c-4584-9e8a-d709799b5730\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") "
Nov 01 13:07:00.531536 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531465    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c552b5a6-2b9c-4584-9e8a-d709799b5730-tls-certs\" (UniqueName: \"kubernetes.io/secret/c552b5a6-2b9c-4584-9e8a-d709799b5730-tls-certs\") pod \"c552b5a6-2b9c-4584-9e8a-d709799b5730\" (UID: \"c552b5a6-2b9c-4584-9e8a-d709799b5730\") "
Nov 01 13:07:00.531536 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531465    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-model-cache" pod "c552b5a6-2b9c-4584-9e8a-d709799b5730" (UID: "c552b5a6-2b9c-4584-9e8a-d709799b5730"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:07:00.531725 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531686    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-home" pod "c552b5a6-2b9c-4584-9e8a-d709799b5730" (UID: "c552b5a6-2b9c-4584-9e8a-d709799b5730"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:07:00.531725 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531698    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-tmp-dir" pod "c552b5a6-2b9c-4584-9e8a-d709799b5730" (UID: "c552b5a6-2b9c-4584-9e8a-d709799b5730"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:07:00.531915 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531888    4095 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-home\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:00.531915 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531909    4095 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-tmp-dir\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:00.532069 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.531924    4095 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-model-cache\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:00.533522 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.533498    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-dshm" pod "c552b5a6-2b9c-4584-9e8a-d709799b5730" (UID: "c552b5a6-2b9c-4584-9e8a-d709799b5730"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:07:00.533618 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.533514    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c552b5a6-2b9c-4584-9e8a-d709799b5730-tls-certs" pod "c552b5a6-2b9c-4584-9e8a-d709799b5730" (UID: "c552b5a6-2b9c-4584-9e8a-d709799b5730"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:07:00.533618 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.533569    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c552b5a6-2b9c-4584-9e8a-d709799b5730-kube-api-access-qmrw8" pod "c552b5a6-2b9c-4584-9e8a-d709799b5730" (UID: "c552b5a6-2b9c-4584-9e8a-d709799b5730"). InnerVolumeSpecName "kube-api-access-qmrw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:07:00.632625 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.632585    4095 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c552b5a6-2b9c-4584-9e8a-d709799b5730-dshm\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:00.632625 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.632616    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmrw8\" (UniqueName: \"kubernetes.io/projected/c552b5a6-2b9c-4584-9e8a-d709799b5730-kube-api-access-qmrw8\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:00.632625 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.632627    4095 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c552b5a6-2b9c-4584-9e8a-d709799b5730-tls-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:07:00.758560 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.758477    4095 generic.go:356] "Generic (PLEG): container finished" podID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerID="a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e" exitCode=0
Nov 01 13:07:00.758560 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.758533    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"
Nov 01 13:07:00.758799 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.758552    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" event={"ID":"c552b5a6-2b9c-4584-9e8a-d709799b5730","Type":"ContainerDied","Data":"a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e"}
Nov 01 13:07:00.758799 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.758604    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw" event={"ID":"c552b5a6-2b9c-4584-9e8a-d709799b5730","Type":"ContainerDied","Data":"4ee43a5bb6fac83833076f9b4c1ea12ea824e6a4ecc0ad26dd72d2fefef3853e"}
Nov 01 13:07:00.758799 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.758626    4095 scope.go:122] "RemoveContainer" containerID="a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e"
Nov 01 13:07:00.768460 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.768444    4095 scope.go:122] "RemoveContainer" containerID="a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e"
Nov 01 13:07:00.768710 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:07:00.768693    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e\": container with ID starting with a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e not found: ID does not exist" containerID="a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e"
Nov 01 13:07:00.768766 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.768719    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e"} err="failed to get container status \"a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e\": rpc error: code = NotFound desc = could not find container \"a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e\": container with ID starting with a24b051511c970b0449bb1c6d0a3b5814ac9c414fb6dc528836ce134f237207e not found: ID does not exist"
Nov 01 13:07:00.916086 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.916052    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"]
Nov 01 13:07:00.918100 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:00.918079    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-f7427d/premium-w6-f7427d-kserve-6c45dc7ccd-7mntw"]
Nov 01 13:07:02.188776 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:02.188741    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" path="/var/lib/kubelet/pods/c552b5a6-2b9c-4584-9e8a-d709799b5730/volumes"
Nov 01 13:07:19.117473 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:07:19.117432    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:07:20.556410 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:20.556381    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:07:20.566008 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:07:20.565986    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:07:49.113918 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:07:49.113871    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:08:17.043626 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:08:17.043592    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"]
Nov 01 13:08:17.090940 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:08:17.090904    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-52b3c667-krgfd"]
Nov 01 13:08:18.187760 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:08:18.187729    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27d8fb21-1d01-453a-b79c-16a851bfb052" path="/var/lib/kubelet/pods/27d8fb21-1d01-453a-b79c-16a851bfb052/volumes"
Nov 01 13:08:19.113442 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:08:19.113407    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:08:21.022291 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:08:21.022260    4095 scope.go:122] "RemoveContainer" containerID="cfda49805ba3f70cacf8d2d6d4c934f6463d971b7501be56efc8b53668eac1d1"
Nov 01 13:08:49.112406 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:08:49.112362    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:09:10.044402 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:09:10.044370    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"]
Nov 01 13:09:10.091039 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:09:10.091013    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6af02137-xrjlc"]
Nov 01 13:09:10.188628 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:09:10.188596    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a" path="/var/lib/kubelet/pods/f2fa8148-9ba9-4cfc-a629-7448cf6c5f8a/volumes"
Nov 01 13:09:19.112998 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:09:19.112944    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:09:21.042715 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:09:21.042687    4095 scope.go:122] "RemoveContainer" containerID="f7f4afb28b31a16ae3da00b50036b8cb53840a1cb6b339982ec73ca8569bf8bd"
Nov 01 13:09:25.044680 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:09:25.044648    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"]
Nov 01 13:09:25.091868 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:09:25.091834    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-937d9115-g4tsl"]
Nov 01 13:09:26.188731 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:09:26.188696    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f48878-03df-4294-b751-95d3517ea79f" path="/var/lib/kubelet/pods/93f48878-03df-4294-b751-95d3517ea79f/volumes"
Nov 01 13:09:49.113138 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:09:49.113096    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:10:02.046029 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:10:02.045991    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"]
Nov 01 13:10:02.047790 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:10:02.047765    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fc7f81ce-wlg9d"]
Nov 01 13:10:02.187861 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:10:02.187828    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf3d8e04-4e53-4345-b458-a61f71a67f4f" path="/var/lib/kubelet/pods/bf3d8e04-4e53-4345-b458-a61f71a67f4f/volumes"
Nov 01 13:10:15.043263 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:10:15.043228    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"]
Nov 01 13:10:15.090320 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:10:15.090290    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a5b3222c-vb95q"]
Nov 01 13:10:16.189286 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:10:16.189254    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b31eaa1-9402-4eef-a81e-9e8dfcff14b7" path="/var/lib/kubelet/pods/0b31eaa1-9402-4eef-a81e-9e8dfcff14b7/volumes"
Nov 01 13:10:19.114361 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:10:19.114325    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:10:21.063280 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:10:21.063250    4095 scope.go:122] "RemoveContainer" containerID="f10ec798419664baaddfce2077f7182c96f5aefc85b49b8a4b2f702fb17cb3d4"
Nov 01 13:10:21.071778 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:10:21.071757    4095 scope.go:122] "RemoveContainer" containerID="93932b579874742b833d1bb8dbd55e3faff4bb8c4c005989b6e9ebfc703ee89d"
Nov 01 13:10:21.079648 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:10:21.079632    4095 scope.go:122] "RemoveContainer" containerID="3ec07766034f918d01987afe8274f134e6c751d5045376060e51356b70a0e7d8"
Nov 01 13:10:49.112471 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:10:49.112427    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:11:19.129717 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:11:19.129643    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:11:21.122995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:11:21.122801    4095 scope.go:122] "RemoveContainer" containerID="62b12926e72a18e66195385c18c3ff20ff68e97ac008beca7382bf782b16d7ef"
Nov 01 13:11:21.165167 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:11:21.162025    4095 scope.go:122] "RemoveContainer" containerID="226fbb954068ccb638b048922ccf220c52edab8fb01e8b804790c311e7f15c79"
Nov 01 13:11:21.219392 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:11:21.219363    4095 scope.go:122] "RemoveContainer" containerID="ee6f056f4c7579a51d570166e782f180c46da61e7cb218cd7c991b04d504672b"
Nov 01 13:11:49.247005 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:11:49.246221    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:12:19.112389 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:12:19.112344    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:12:20.589075 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:12:20.589047    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:12:20.599366 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:12:20.599346    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:12:21.345256 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:12:21.345224    4095 scope.go:122] "RemoveContainer" containerID="c7236884dd6688879cc7afe7cdc101a28e81d0c5538e1778715989e8845c7586"
Nov 01 13:12:49.127037 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:12:49.126936    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:13:19.113649 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:13:19.113608    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:13:49.126989 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:13:49.126849    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:14:09.923995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:14:09.923078    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gjsds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 13:14:09.923995 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:14:09.923163    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" podUID="490cb0f8-5280-4505-9141-43cd22eb0cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:14:19.115187 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:14:19.115142    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:14:49.111471 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:14:49.111427    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:15:04.291117 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291076    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"]
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291527    4095 cpu_manager.go:414] "containerMap: removing container" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291539    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291548    4095 cpu_manager.go:414] "containerMap: removing container" podUID="5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5" containerName="revoke-keys"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291553    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5" containerName="revoke-keys"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291566    4095 cpu_manager.go:414] "containerMap: removing container" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291571    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291585    4095 cpu_manager.go:414] "containerMap: removing container" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291590    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291657    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291666    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="cec0037a-c467-4cb4-bb5b-d556ff5e7cf6" containerName="istio-proxy"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291679    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="c552b5a6-2b9c-4584-9e8a-d709799b5730" containerName="main"
Nov 01 13:15:04.293519 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.291685    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a63e1b9-fb6e-471d-8f3f-baeb8c1898a5" containerName="revoke-keys"
Nov 01 13:15:04.294550 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.294536    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.344943 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.344910    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-483de8-serving-cert\""
Nov 01 13:15:04.394100 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.394063    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"]
Nov 01 13:15:04.486916 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.486879    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.487098 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.486931    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.487098 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.487078    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqjtn\" (UniqueName: \"kubernetes.io/projected/e92df2ec-f35b-401e-bcfa-f913ef40944d-kube-api-access-mqjtn\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.487212 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.487177    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92df2ec-f35b-401e-bcfa-f913ef40944d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.588004 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.587897    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.588004 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.587937    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.588213 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.588019    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqjtn\" (UniqueName: \"kubernetes.io/projected/e92df2ec-f35b-401e-bcfa-f913ef40944d-kube-api-access-mqjtn\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.588213 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.588085    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92df2ec-f35b-401e-bcfa-f913ef40944d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.588773 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.588747    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92df2ec-f35b-401e-bcfa-f913ef40944d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.590719 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.590695    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.590946 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.590927    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.642683 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.642647    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqjtn\" (UniqueName: \"kubernetes.io/projected/e92df2ec-f35b-401e-bcfa-f913ef40944d-kube-api-access-mqjtn\") pod \"maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:04.907438 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:04.907400    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:05.273875 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:05.273845    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"]
Nov 01 13:15:05.275424 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:15:05.275395    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode92df2ec_f35b_401e_bcfa_f913ef40944d.slice/crio-47967bc1b4f706ea987de814f762541611358a62bda7d159197c85ef0038eac2 WatchSource:0}: Error finding container 47967bc1b4f706ea987de814f762541611358a62bda7d159197c85ef0038eac2: Status 404 returned error can't find the container with id 47967bc1b4f706ea987de814f762541611358a62bda7d159197c85ef0038eac2
Nov 01 13:15:05.276983 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:05.276956    4095 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 13:15:05.598805 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:05.598721    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b" event={"ID":"e92df2ec-f35b-401e-bcfa-f913ef40944d","Type":"ContainerStarted","Data":"47967bc1b4f706ea987de814f762541611358a62bda7d159197c85ef0038eac2"}
Nov 01 13:15:07.609928 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:07.609890    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b" event={"ID":"e92df2ec-f35b-401e-bcfa-f913ef40944d","Type":"ContainerStarted","Data":"3b91ce237942374e60cd85688c10dc2f4aaa9e9a4d35ff5fc2382825780d98bf"}
Nov 01 13:15:07.610312 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:07.610141    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:07.758625 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:07.758564    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b" podStartSLOduration=1.4994470770000001 podStartE2EDuration="3.758551704s" totalImagesPullingTime="2.259104627s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:15:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:15:07.758349728 +0000 UTC m=+3167.676145396" watchObservedRunningTime="2026-11-01 13:15:07.758551704 +0000 UTC m=+3167.676347365"
Nov 01 13:15:13.619393 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:15:13.619363    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:15:19.113056 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:15:19.113021    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:15:49.113390 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:15:49.113309    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:16:19.112016 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:16:19.111960    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:16:33.898304 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:33.898270    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"]
Nov 01 13:16:33.898889 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:33.898512    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b" podUID="e92df2ec-f35b-401e-bcfa-f913ef40944d" containerName="maas-api" containerID="cri-o://3b91ce237942374e60cd85688c10dc2f4aaa9e9a4d35ff5fc2382825780d98bf" gracePeriod=30
Nov 01 13:16:35.053307 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.053273    4095 generic.go:356] "Generic (PLEG): container finished" podID="e92df2ec-f35b-401e-bcfa-f913ef40944d" containerID="3b91ce237942374e60cd85688c10dc2f4aaa9e9a4d35ff5fc2382825780d98bf" exitCode=0
Nov 01 13:16:35.053708 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.053345    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b" event={"ID":"e92df2ec-f35b-401e-bcfa-f913ef40944d","Type":"ContainerDied","Data":"3b91ce237942374e60cd85688c10dc2f4aaa9e9a4d35ff5fc2382825780d98bf"}
Nov 01 13:16:35.157857 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.157830    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:16:35.234203 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.234134    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-maas-api-tls\") pod \"e92df2ec-f35b-401e-bcfa-f913ef40944d\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") "
Nov 01 13:16:35.234203 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.234189    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e92df2ec-f35b-401e-bcfa-f913ef40944d-kube-api-access-mqjtn\" (UniqueName: \"kubernetes.io/projected/e92df2ec-f35b-401e-bcfa-f913ef40944d-kube-api-access-mqjtn\") pod \"e92df2ec-f35b-401e-bcfa-f913ef40944d\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") "
Nov 01 13:16:35.234403 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.234223    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-metrics-certs\") pod \"e92df2ec-f35b-401e-bcfa-f913ef40944d\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") "
Nov 01 13:16:35.234403 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.234281    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e92df2ec-f35b-401e-bcfa-f913ef40944d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92df2ec-f35b-401e-bcfa-f913ef40944d-openshift-service-ca\") pod \"e92df2ec-f35b-401e-bcfa-f913ef40944d\" (UID: \"e92df2ec-f35b-401e-bcfa-f913ef40944d\") "
Nov 01 13:16:35.234716 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.234691    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e92df2ec-f35b-401e-bcfa-f913ef40944d-openshift-service-ca" pod "e92df2ec-f35b-401e-bcfa-f913ef40944d" (UID: "e92df2ec-f35b-401e-bcfa-f913ef40944d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:16:35.236324 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.236299    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-maas-api-tls" pod "e92df2ec-f35b-401e-bcfa-f913ef40944d" (UID: "e92df2ec-f35b-401e-bcfa-f913ef40944d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:16:35.236416 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.236369    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e92df2ec-f35b-401e-bcfa-f913ef40944d-kube-api-access-mqjtn" pod "e92df2ec-f35b-401e-bcfa-f913ef40944d" (UID: "e92df2ec-f35b-401e-bcfa-f913ef40944d"). InnerVolumeSpecName "kube-api-access-mqjtn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:16:35.236416 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.236380    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-metrics-certs" pod "e92df2ec-f35b-401e-bcfa-f913ef40944d" (UID: "e92df2ec-f35b-401e-bcfa-f913ef40944d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:16:35.336012 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.335958    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:35.336012 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.336011    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqjtn\" (UniqueName: \"kubernetes.io/projected/e92df2ec-f35b-401e-bcfa-f913ef40944d-kube-api-access-mqjtn\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:35.336212 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.336023    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e92df2ec-f35b-401e-bcfa-f913ef40944d-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:35.336212 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:35.336037    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e92df2ec-f35b-401e-bcfa-f913ef40944d-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:16:36.059178 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:36.059147    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"
Nov 01 13:16:36.059634 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:36.059175    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b" event={"ID":"e92df2ec-f35b-401e-bcfa-f913ef40944d","Type":"ContainerDied","Data":"47967bc1b4f706ea987de814f762541611358a62bda7d159197c85ef0038eac2"}
Nov 01 13:16:36.059634 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:36.059238    4095 scope.go:122] "RemoveContainer" containerID="3b91ce237942374e60cd85688c10dc2f4aaa9e9a4d35ff5fc2382825780d98bf"
Nov 01 13:16:36.220533 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:36.220498    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"]
Nov 01 13:16:36.268117 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:36.268080    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-483de8-5bbccdfb96-gjt4b"]
Nov 01 13:16:38.190536 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:16:38.190496    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e92df2ec-f35b-401e-bcfa-f913ef40944d" path="/var/lib/kubelet/pods/e92df2ec-f35b-401e-bcfa-f913ef40944d/volumes"
Nov 01 13:16:49.113571 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:16:49.113532    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:17:19.113447 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:17:19.113407    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:17:20.621244 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:17:20.621210    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:17:20.632159 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:17:20.632139    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:17:49.114417 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:17:49.114377    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:18:19.113053 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:18:19.113012    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:18:49.197516 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:18:49.197221    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:18:49.965001 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:18:49.964180    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-9m98v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 13:18:49.965001 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:18:49.964276    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-9m98v" podUID="f3566191-2934-4500-90e2-c12dbffb0354" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:19:19.113445 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:19:19.113407    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:19:42.691181 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.691149    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"]
Nov 01 13:19:42.691674 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.691605    4095 cpu_manager.go:414] "containerMap: removing container" podUID="e92df2ec-f35b-401e-bcfa-f913ef40944d" containerName="maas-api"
Nov 01 13:19:42.691674 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.691615    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e92df2ec-f35b-401e-bcfa-f913ef40944d" containerName="maas-api"
Nov 01 13:19:42.691754 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.691700    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="e92df2ec-f35b-401e-bcfa-f913ef40944d" containerName="maas-api"
Nov 01 13:19:42.694820 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.694800    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.744747 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.744721    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f38349-serving-cert\""
Nov 01 13:19:42.753295 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.753268    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e144389-ad33-4fd0-a05e-fdc0d496a704-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.753432 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.753331    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.753477 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.753441    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v62x7\" (UniqueName: \"kubernetes.io/projected/2e144389-ad33-4fd0-a05e-fdc0d496a704-kube-api-access-v62x7\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.753551 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.753530    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.794360 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.794332    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"]
Nov 01 13:19:42.854428 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.854396    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.854615 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.854510    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e144389-ad33-4fd0-a05e-fdc0d496a704-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.854615 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.854606    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.854755 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.854649    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v62x7\" (UniqueName: \"kubernetes.io/projected/2e144389-ad33-4fd0-a05e-fdc0d496a704-kube-api-access-v62x7\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.854813 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:19:42.854753    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-serving-cert: secret "maas-api-e2e-worker-single-main-f38349-serving-cert" not found
Nov 01 13:19:42.854867 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:19:42.854831    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls podName:2e144389-ad33-4fd0-a05e-fdc0d496a704 nodeName:}" failed. No retries permitted until 2026-11-01 13:19:43.354805414 +0000 UTC m=+3443.272601092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls") pod "maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s" (UID: "2e144389-ad33-4fd0-a05e-fdc0d496a704") : secret "maas-api-e2e-worker-single-main-f38349-serving-cert" not found
Nov 01 13:19:42.855244 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.855222    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e144389-ad33-4fd0-a05e-fdc0d496a704-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.856877 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.856852    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:42.911553 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:42.911511    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v62x7\" (UniqueName: \"kubernetes.io/projected/2e144389-ad33-4fd0-a05e-fdc0d496a704-kube-api-access-v62x7\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:43.360504 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:43.360469    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:43.362732 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:43.362712    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:43.608386 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:43.608343    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:43.972761 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:43.972731    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"]
Nov 01 13:19:43.975109 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:19:43.975079    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e144389_ad33_4fd0_a05e_fdc0d496a704.slice/crio-b77cbacd2e572034dffff843ede37f6759907963057e1058b6736cb0913d70d4 WatchSource:0}: Error finding container b77cbacd2e572034dffff843ede37f6759907963057e1058b6736cb0913d70d4: Status 404 returned error can't find the container with id b77cbacd2e572034dffff843ede37f6759907963057e1058b6736cb0913d70d4
Nov 01 13:19:44.039203 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:44.039166    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s" event={"ID":"2e144389-ad33-4fd0-a05e-fdc0d496a704","Type":"ContainerStarted","Data":"b77cbacd2e572034dffff843ede37f6759907963057e1058b6736cb0913d70d4"}
Nov 01 13:19:45.052812 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:45.052753    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s" event={"ID":"2e144389-ad33-4fd0-a05e-fdc0d496a704","Type":"ContainerStarted","Data":"e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13"}
Nov 01 13:19:45.053229 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:45.052864    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:19:45.186630 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:45.186532    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s" podStartSLOduration=2.251703942 podStartE2EDuration="3.186518479s" totalImagesPullingTime="934.814537ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:19:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:19:45.18504393 +0000 UTC m=+3445.102839596" watchObservedRunningTime="2026-11-01 13:19:45.186518479 +0000 UTC m=+3445.104314165"
Nov 01 13:19:49.114805 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:19:49.114765    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:19:51.063452 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:19:51.063423    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:20:19.113489 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:20:19.113447    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:20:49.112719 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:20:49.112680    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:21:12.406911 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:12.406880    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"]
Nov 01 13:21:12.407436 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:12.407099    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s" podUID="2e144389-ad33-4fd0-a05e-fdc0d496a704" containerName="maas-api" containerID="cri-o://e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13" gracePeriod=30
Nov 01 13:21:13.655621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.655598    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:21:13.699742 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.699717    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls\") pod \"2e144389-ad33-4fd0-a05e-fdc0d496a704\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") "
Nov 01 13:21:13.699901 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.699768    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e144389-ad33-4fd0-a05e-fdc0d496a704-kube-api-access-v62x7\" (UniqueName: \"kubernetes.io/projected/2e144389-ad33-4fd0-a05e-fdc0d496a704-kube-api-access-v62x7\") pod \"2e144389-ad33-4fd0-a05e-fdc0d496a704\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") "
Nov 01 13:21:13.700051 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.700026    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-metrics-certs\") pod \"2e144389-ad33-4fd0-a05e-fdc0d496a704\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") "
Nov 01 13:21:13.700143 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.700072    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e144389-ad33-4fd0-a05e-fdc0d496a704-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e144389-ad33-4fd0-a05e-fdc0d496a704-openshift-service-ca\") pod \"2e144389-ad33-4fd0-a05e-fdc0d496a704\" (UID: \"2e144389-ad33-4fd0-a05e-fdc0d496a704\") "
Nov 01 13:21:13.700430 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.700410    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e144389-ad33-4fd0-a05e-fdc0d496a704-openshift-service-ca" pod "2e144389-ad33-4fd0-a05e-fdc0d496a704" (UID: "2e144389-ad33-4fd0-a05e-fdc0d496a704"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:21:13.702002 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.701962    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-metrics-certs" pod "2e144389-ad33-4fd0-a05e-fdc0d496a704" (UID: "2e144389-ad33-4fd0-a05e-fdc0d496a704"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:21:13.702278 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.702250    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls" pod "2e144389-ad33-4fd0-a05e-fdc0d496a704" (UID: "2e144389-ad33-4fd0-a05e-fdc0d496a704"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:21:13.702356 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.702311    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e144389-ad33-4fd0-a05e-fdc0d496a704-kube-api-access-v62x7" pod "2e144389-ad33-4fd0-a05e-fdc0d496a704" (UID: "2e144389-ad33-4fd0-a05e-fdc0d496a704"). InnerVolumeSpecName "kube-api-access-v62x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:21:13.801980 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.801886    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:13.801980 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.801917    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e144389-ad33-4fd0-a05e-fdc0d496a704-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:13.801980 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.801929    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2e144389-ad33-4fd0-a05e-fdc0d496a704-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:13.801980 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:13.801938    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v62x7\" (UniqueName: \"kubernetes.io/projected/2e144389-ad33-4fd0-a05e-fdc0d496a704-kube-api-access-v62x7\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:21:14.480955 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:14.480872    4095 generic.go:356] "Generic (PLEG): container finished" podID="2e144389-ad33-4fd0-a05e-fdc0d496a704" containerID="e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13" exitCode=0
Nov 01 13:21:14.480955 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:14.480919    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"
Nov 01 13:21:14.480955 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:14.480945    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s" event={"ID":"2e144389-ad33-4fd0-a05e-fdc0d496a704","Type":"ContainerDied","Data":"e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13"}
Nov 01 13:21:14.481241 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:14.481002    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s" event={"ID":"2e144389-ad33-4fd0-a05e-fdc0d496a704","Type":"ContainerDied","Data":"b77cbacd2e572034dffff843ede37f6759907963057e1058b6736cb0913d70d4"}
Nov 01 13:21:14.481241 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:14.481022    4095 scope.go:122] "RemoveContainer" containerID="e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13"
Nov 01 13:21:14.490245 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:14.490217    4095 scope.go:122] "RemoveContainer" containerID="e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13"
Nov 01 13:21:14.493280 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:21:14.493249    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13\": container with ID starting with e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13 not found: ID does not exist" containerID="e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13"
Nov 01 13:21:14.493383 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:14.493294    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13"} err="failed to get container status \"e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13\": rpc error: code = NotFound desc = could not find container \"e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13\": container with ID starting with e45e5ef29c6442ae2e899ffa161c3de0f9c072d3a4dcbc2619b2d51fe8674c13 not found: ID does not exist"
Nov 01 13:21:14.636156 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:14.636117    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"]
Nov 01 13:21:14.683223 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:14.683191    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f38349-68d6957d59-hb22s"]
Nov 01 13:21:16.187986 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:21:16.187926    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e144389-ad33-4fd0-a05e-fdc0d496a704" path="/var/lib/kubelet/pods/2e144389-ad33-4fd0-a05e-fdc0d496a704/volumes"
Nov 01 13:21:19.114726 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:21:19.114693    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:21:49.112801 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:21:49.112760    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:22:19.113158 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:22:19.113121    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:22:20.655378 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:22:20.655347    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:22:20.668525 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:22:20.668505    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:22:49.112706 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:22:49.112666    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:23:09.914994 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:23:09.914076    4095 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-gjsds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 01 13:23:09.914994 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:23:09.914157    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-gjsds" podUID="490cb0f8-5280-4505-9141-43cd22eb0cac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 01 13:23:19.111936 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:23:19.111894    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:23:49.112365 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:23:49.112331    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:24:03.419061 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.419024    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"]
Nov 01 13:24:03.419540 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.419514    4095 cpu_manager.go:414] "containerMap: removing container" podUID="2e144389-ad33-4fd0-a05e-fdc0d496a704" containerName="maas-api"
Nov 01 13:24:03.419540 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.419527    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e144389-ad33-4fd0-a05e-fdc0d496a704" containerName="maas-api"
Nov 01 13:24:03.419643 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.419630    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e144389-ad33-4fd0-a05e-fdc0d496a704" containerName="maas-api"
Nov 01 13:24:03.422762 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.422745    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.472452 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.472415    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f3c81bef-openshift-default-dockercfg-s9qgd\""
Nov 01 13:24:03.522697 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.522667    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"]
Nov 01 13:24:03.551454 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.551423    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-certs\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.551454 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.551456    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-data\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.551694 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.551483    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k2rgl\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-kube-api-access-k2rgl\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.551694 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.551581    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-credential-socket\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.551831 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.551706    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-envoy\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.551831 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.551766    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istiod-ca-cert\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.551910 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.551858    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-podinfo\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.551955 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.551927    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-token\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.552058 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.552038    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-socket\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.653420 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653385    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-envoy\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.653620 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653433    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istiod-ca-cert\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.653620 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653461    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-podinfo\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.653620 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653498    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-token\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.653620 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653526    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-socket\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.653620 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653586    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-certs\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.653620 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653612    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-data\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.653953 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653647    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k2rgl\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-kube-api-access-k2rgl\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.653953 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653675    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-credential-socket\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.654098 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.653943    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-socket\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.654098 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.654071    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-credential-socket\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.654265 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.654240    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-certs\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.654333 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.654280    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-data\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.654395 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.654334    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istiod-ca-cert\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.655778 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.655750    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-envoy\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.655981 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.655951    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-podinfo\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.707947 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.707877    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-token\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.708118 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.708096    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k2rgl\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-kube-api-access-k2rgl\") pod \"e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") " pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:03.736269 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:03.736238    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:04.152413 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.152383    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"]
Nov 01 13:24:04.153538 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:24:04.153510    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf51f9dbb_0870_4cb1_8ddc_d440a86ee112.slice/crio-5f61e2059e568273f337fc8a981af789ff2f5d1887100536f3fcab8376322029 WatchSource:0}: Error finding container 5f61e2059e568273f337fc8a981af789ff2f5d1887100536f3fcab8376322029: Status 404 returned error can't find the container with id 5f61e2059e568273f337fc8a981af789ff2f5d1887100536f3fcab8376322029
Nov 01 13:24:04.155848 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.155817    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:24:04.155966 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.155891    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:24:04.155966 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.155938    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:24:04.502116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.502014    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" event={"ID":"f51f9dbb-0870-4cb1-8ddc-d440a86ee112","Type":"ContainerStarted","Data":"f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d"}
Nov 01 13:24:04.502582 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.502116    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" event={"ID":"f51f9dbb-0870-4cb1-8ddc-d440a86ee112","Type":"ContainerStarted","Data":"5f61e2059e568273f337fc8a981af789ff2f5d1887100536f3fcab8376322029"}
Nov 01 13:24:04.633806 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.633754    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" podStartSLOduration=1.633741587 podStartE2EDuration="1.633741587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:24:04.6314584 +0000 UTC m=+3704.549254067" watchObservedRunningTime="2026-11-01 13:24:04.633741587 +0000 UTC m=+3704.551537253"
Nov 01 13:24:04.737416 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.737379    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:04.790941 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.790863    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6985888875-4bv96"]
Nov 01 13:24:04.794666 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.794640    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.850418 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.850383    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6985888875-4bv96"]
Nov 01 13:24:04.866580 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.866545    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54ph4\" (UniqueName: \"kubernetes.io/projected/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-kube-api-access-54ph4\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.866803 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.866779    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-maas-api-tls\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.866878 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.866822    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-openshift-service-ca\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.866954 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.866932    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-metrics-certs\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.968098 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.968060    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-maas-api-tls\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.968329 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.968112    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-openshift-service-ca\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.968329 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.968154    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-metrics-certs\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.968329 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.968225    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54ph4\" (UniqueName: \"kubernetes.io/projected/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-kube-api-access-54ph4\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.968924 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.968895    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-openshift-service-ca\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.970536 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.970518    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-metrics-certs\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:04.970692 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:04.970674    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-maas-api-tls\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:05.031044 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:05.031016    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54ph4\" (UniqueName: \"kubernetes.io/projected/4d6b5e9a-f783-45b8-8a1f-d88ac3810de7-kube-api-access-54ph4\") pod \"maas-api-6985888875-4bv96\" (UID: \"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7\") " pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:05.112164 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:05.112135    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:05.484457 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:05.484431    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6985888875-4bv96"]
Nov 01 13:24:05.486174 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:24:05.486150    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6b5e9a_f783_45b8_8a1f_d88ac3810de7.slice/crio-a915d9950298489a338307f96d0610fe6c2b2bd5896414cbc8be46ab13e414b8 WatchSource:0}: Error finding container a915d9950298489a338307f96d0610fe6c2b2bd5896414cbc8be46ab13e414b8: Status 404 returned error can't find the container with id a915d9950298489a338307f96d0610fe6c2b2bd5896414cbc8be46ab13e414b8
Nov 01 13:24:05.487602 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:05.487582    4095 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 01 13:24:05.510939 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:05.510904    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96" event={"ID":"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7","Type":"ContainerStarted","Data":"a915d9950298489a338307f96d0610fe6c2b2bd5896414cbc8be46ab13e414b8"}
Nov 01 13:24:05.741751 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:05.741678    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:06.520101 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:06.520064    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96" event={"ID":"4d6b5e9a-f783-45b8-8a1f-d88ac3810de7","Type":"ContainerStarted","Data":"d3958dcb1a0668e6736462f1bf93a536c8fb7598f7a118325e3e0305b6d2d9a6"}
Nov 01 13:24:06.520549 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:06.520256    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:06.521399 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:06.521375    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:06.629759 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:06.629710    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96" podStartSLOduration=1.731759307 podStartE2EDuration="2.629696497s" totalImagesPullingTime="897.93719ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:24:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:24:06.629187338 +0000 UTC m=+3706.546983025" watchObservedRunningTime="2026-11-01 13:24:06.629696497 +0000 UTC m=+3706.547492165"
Nov 01 13:24:07.524930 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:07.524899    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:13.534911 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:13.534881    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6985888875-4bv96"
Nov 01 13:24:13.694879 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:13.694846    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"]
Nov 01 13:24:13.695134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:13.695108    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl" podUID="1f347965-a09d-486c-bfa8-fa9cbe28e531" containerName="maas-api" containerID="cri-o://d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1" gracePeriod=30
Nov 01 13:24:14.951463 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:14.951438    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 13:24:15.078694 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.078609    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f347965-a09d-486c-bfa8-fa9cbe28e531-kube-api-access-krjsp\" (UniqueName: \"kubernetes.io/projected/1f347965-a09d-486c-bfa8-fa9cbe28e531-kube-api-access-krjsp\") pod \"1f347965-a09d-486c-bfa8-fa9cbe28e531\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") "
Nov 01 13:24:15.078694 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.078680    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-maas-api-tls\") pod \"1f347965-a09d-486c-bfa8-fa9cbe28e531\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") "
Nov 01 13:24:15.078902 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.078716    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-metrics-certs\") pod \"1f347965-a09d-486c-bfa8-fa9cbe28e531\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") "
Nov 01 13:24:15.078902 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.078737    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f347965-a09d-486c-bfa8-fa9cbe28e531-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f347965-a09d-486c-bfa8-fa9cbe28e531-openshift-service-ca\") pod \"1f347965-a09d-486c-bfa8-fa9cbe28e531\" (UID: \"1f347965-a09d-486c-bfa8-fa9cbe28e531\") "
Nov 01 13:24:15.079182 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.079158    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f347965-a09d-486c-bfa8-fa9cbe28e531-openshift-service-ca" pod "1f347965-a09d-486c-bfa8-fa9cbe28e531" (UID: "1f347965-a09d-486c-bfa8-fa9cbe28e531"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:24:15.080895 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.080867    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-maas-api-tls" pod "1f347965-a09d-486c-bfa8-fa9cbe28e531" (UID: "1f347965-a09d-486c-bfa8-fa9cbe28e531"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:24:15.081185 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.081164    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-metrics-certs" pod "1f347965-a09d-486c-bfa8-fa9cbe28e531" (UID: "1f347965-a09d-486c-bfa8-fa9cbe28e531"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:24:15.081185 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.081172    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f347965-a09d-486c-bfa8-fa9cbe28e531-kube-api-access-krjsp" pod "1f347965-a09d-486c-bfa8-fa9cbe28e531" (UID: "1f347965-a09d-486c-bfa8-fa9cbe28e531"). InnerVolumeSpecName "kube-api-access-krjsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:24:15.179812 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.179766    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krjsp\" (UniqueName: \"kubernetes.io/projected/1f347965-a09d-486c-bfa8-fa9cbe28e531-kube-api-access-krjsp\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:24:15.179812 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.179810    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:24:15.179812 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.179821    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f347965-a09d-486c-bfa8-fa9cbe28e531-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:24:15.180064 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.179831    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f347965-a09d-486c-bfa8-fa9cbe28e531-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:24:15.564790 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.564758    4095 generic.go:356] "Generic (PLEG): container finished" podID="1f347965-a09d-486c-bfa8-fa9cbe28e531" containerID="d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1" exitCode=0
Nov 01 13:24:15.565020 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.564807    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"
Nov 01 13:24:15.565020 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.564839    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl" event={"ID":"1f347965-a09d-486c-bfa8-fa9cbe28e531","Type":"ContainerDied","Data":"d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1"}
Nov 01 13:24:15.565020 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.564905    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl" event={"ID":"1f347965-a09d-486c-bfa8-fa9cbe28e531","Type":"ContainerDied","Data":"adac5a9d0b9f93586f59392481570ce656b1e840be6d2a799b44249c63312875"}
Nov 01 13:24:15.565020 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.564936    4095 scope.go:122] "RemoveContainer" containerID="d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1"
Nov 01 13:24:15.575026 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.575011    4095 scope.go:122] "RemoveContainer" containerID="d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1"
Nov 01 13:24:15.575295 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:24:15.575277    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1\": container with ID starting with d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1 not found: ID does not exist" containerID="d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1"
Nov 01 13:24:15.575341 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.575305    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1"} err="failed to get container status \"d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1\": rpc error: code = NotFound desc = could not find container \"d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1\": container with ID starting with d4d5d06e5c5f8a88cd26f5e2cfec71290f26547b9f2b9298b1fe4fd7ddfb66c1 not found: ID does not exist"
Nov 01 13:24:15.719613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.719577    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"]
Nov 01 13:24:15.723659 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:15.723207    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cdfff487-4z6sl"]
Nov 01 13:24:16.188199 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:16.188167    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f347965-a09d-486c-bfa8-fa9cbe28e531" path="/var/lib/kubelet/pods/1f347965-a09d-486c-bfa8-fa9cbe28e531/volumes"
Nov 01 13:24:19.136899 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:24:19.136863    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:24:27.640548 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:27.640521    4095 generic.go:356] "Generic (PLEG): container finished" podID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerID="f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d" exitCode=0
Nov 01 13:24:27.640922 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:27.640591    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" event={"ID":"f51f9dbb-0870-4cb1-8ddc-d440a86ee112","Type":"ContainerDied","Data":"f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d"}
Nov 01 13:24:27.640964 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:27.640927    4095 scope.go:122] "RemoveContainer" containerID="f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d"
Nov 01 13:24:27.642002 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:27.641952    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:24:27.642118 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:27.642029    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:24:27.642118 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:27.642062    4095 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236076Ki","pods":"250"}
Nov 01 13:24:28.647563 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:28.647531    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" event={"ID":"f51f9dbb-0870-4cb1-8ddc-d440a86ee112","Type":"ContainerStarted","Data":"3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7"}
Nov 01 13:24:28.736633 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:28.736600    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:28.736809 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:28.736772    4095 patch_prober.go:29] interesting pod/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 13:24:28.736851 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:28.736830    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 13:24:29.737652 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:29.737611    4095 patch_prober.go:29] interesting pod/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 13:24:29.738134 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:29.737668    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 13:24:30.737099 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:30.737063    4095 patch_prober.go:29] interesting pod/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 13:24:30.737295 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:30.737118    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 13:24:31.736676 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:31.736639    4095 patch_prober.go:29] interesting pod/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 13:24:31.737191 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:31.736698    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 13:24:32.736924 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:32.736887    4095 patch_prober.go:29] interesting pod/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body=
Nov 01 13:24:32.737368 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:32.736997    4095 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused"
Nov 01 13:24:33.741206 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:33.741173    4095 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:33.741573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:33.741543    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:33.742290 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:33.742275    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:24:49.113599 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:24:49.113558    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:24:58.993247 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:58.993217    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"]
Nov 01 13:24:58.993662 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:24:58.993448    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy" containerID="cri-o://3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7" gracePeriod=30
Nov 01 13:25:03.743514 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:03.743482    4095 patch_prober.go:29] interesting pod/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 01 13:25:03.743878 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:03.743546    4095 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 01 13:25:04.252429 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.252394    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:25:04.320487 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320396    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-certs\") pod \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") "
Nov 01 13:25:04.320487 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320458    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-credential-socket\") pod \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") "
Nov 01 13:25:04.320724 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320490    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-data\") pod \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") "
Nov 01 13:25:04.320724 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320534    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-kube-api-access-k2rgl\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-kube-api-access-k2rgl\") pod \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") "
Nov 01 13:25:04.320724 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320599    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-podinfo\") pod \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") "
Nov 01 13:25:04.320724 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320627    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-token\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-token\") pod \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") "
Nov 01 13:25:04.320942 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320705    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-certs" pod "f51f9dbb-0870-4cb1-8ddc-d440a86ee112" (UID: "f51f9dbb-0870-4cb1-8ddc-d440a86ee112"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:25:04.320942 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320741    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-envoy\") pod \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") "
Nov 01 13:25:04.320942 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320794    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-data" pod "f51f9dbb-0870-4cb1-8ddc-d440a86ee112" (UID: "f51f9dbb-0870-4cb1-8ddc-d440a86ee112"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:25:04.320942 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320820    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-credential-socket" pod "f51f9dbb-0870-4cb1-8ddc-d440a86ee112" (UID: "f51f9dbb-0870-4cb1-8ddc-d440a86ee112"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:25:04.320942 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320846    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istiod-ca-cert\") pod \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") "
Nov 01 13:25:04.320942 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.320913    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-socket\") pod \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\" (UID: \"f51f9dbb-0870-4cb1-8ddc-d440a86ee112\") "
Nov 01 13:25:04.321268 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.321166    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-socket" pod "f51f9dbb-0870-4cb1-8ddc-d440a86ee112" (UID: "f51f9dbb-0870-4cb1-8ddc-d440a86ee112"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:25:04.321268 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.321181    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istiod-ca-cert" pod "f51f9dbb-0870-4cb1-8ddc-d440a86ee112" (UID: "f51f9dbb-0870-4cb1-8ddc-d440a86ee112"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:25:04.321403 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.321384    4095 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istiod-ca-cert\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:25:04.321470 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.321410    4095 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:25:04.321470 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.321425    4095 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-workload-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:25:04.321470 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.321438    4095 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-credential-socket\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:25:04.321470 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.321452    4095 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-data\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:25:04.323579 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.323552    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-kube-api-access-k2rgl" pod "f51f9dbb-0870-4cb1-8ddc-d440a86ee112" (UID: "f51f9dbb-0870-4cb1-8ddc-d440a86ee112"). InnerVolumeSpecName "kube-api-access-k2rgl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:25:04.323579 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.323570    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-token" pod "f51f9dbb-0870-4cb1-8ddc-d440a86ee112" (UID: "f51f9dbb-0870-4cb1-8ddc-d440a86ee112"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:25:04.323734 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.323569    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-podinfo" pod "f51f9dbb-0870-4cb1-8ddc-d440a86ee112" (UID: "f51f9dbb-0870-4cb1-8ddc-d440a86ee112"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 01 13:25:04.323734 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.323647    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-envoy" pod "f51f9dbb-0870-4cb1-8ddc-d440a86ee112" (UID: "f51f9dbb-0870-4cb1-8ddc-d440a86ee112"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 01 13:25:04.422356 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.422322    4095 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-podinfo\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:25:04.422356 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.422355    4095 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-token\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:25:04.422580 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.422368    4095 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-istio-envoy\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:25:04.422580 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.422380    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k2rgl\" (UniqueName: \"kubernetes.io/projected/f51f9dbb-0870-4cb1-8ddc-d440a86ee112-kube-api-access-k2rgl\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:25:04.825617 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.825585    4095 generic.go:356] "Generic (PLEG): container finished" podID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerID="3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7" exitCode=0
Nov 01 13:25:04.826085 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.825622    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" event={"ID":"f51f9dbb-0870-4cb1-8ddc-d440a86ee112","Type":"ContainerDied","Data":"3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7"}
Nov 01 13:25:04.826085 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.825678    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc" event={"ID":"f51f9dbb-0870-4cb1-8ddc-d440a86ee112","Type":"ContainerDied","Data":"5f61e2059e568273f337fc8a981af789ff2f5d1887100536f3fcab8376322029"}
Nov 01 13:25:04.826085 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.825635    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"
Nov 01 13:25:04.826085 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.825709    4095 scope.go:122] "RemoveContainer" containerID="3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7"
Nov 01 13:25:04.835806 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.835791    4095 scope.go:122] "RemoveContainer" containerID="f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d"
Nov 01 13:25:04.844291 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.844275    4095 scope.go:122] "RemoveContainer" containerID="3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7"
Nov 01 13:25:04.844554 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:25:04.844532    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7\": container with ID starting with 3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7 not found: ID does not exist" containerID="3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7"
Nov 01 13:25:04.844606 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.844561    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7"} err="failed to get container status \"3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7\": rpc error: code = NotFound desc = could not find container \"3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7\": container with ID starting with 3a942c273133a2163c9f711352855ca359ae56bbb99cb8c3f915e1b16796aeb7 not found: ID does not exist"
Nov 01 13:25:04.844606 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.844580    4095 scope.go:122] "RemoveContainer" containerID="f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d"
Nov 01 13:25:04.844808 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:25:04.844791    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d\": container with ID starting with f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d not found: ID does not exist" containerID="f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d"
Nov 01 13:25:04.844867 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.844818    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d"} err="failed to get container status \"f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d\": rpc error: code = NotFound desc = could not find container \"f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d\": container with ID starting with f7be26b3775498ddeb666cf9478c59bccce61550eef887c60b831264a2c19c8d not found: ID does not exist"
Nov 01 13:25:04.991536 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:04.991504    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"]
Nov 01 13:25:05.038469 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:05.038437    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f3c81bef-openshift-default-78568694df-gq5nc"]
Nov 01 13:25:06.189373 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:06.189333    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" path="/var/lib/kubelet/pods/f51f9dbb-0870-4cb1-8ddc-d440a86ee112/volumes"
Nov 01 13:25:08.104639 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.104601    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"]
Nov 01 13:25:08.105185 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.105168    4095 cpu_manager.go:414] "containerMap: removing container" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy"
Nov 01 13:25:08.105252 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.105187    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy"
Nov 01 13:25:08.105252 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.105213    4095 cpu_manager.go:414] "containerMap: removing container" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy"
Nov 01 13:25:08.105252 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.105220    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy"
Nov 01 13:25:08.105252 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.105245    4095 cpu_manager.go:414] "containerMap: removing container" podUID="1f347965-a09d-486c-bfa8-fa9cbe28e531" containerName="maas-api"
Nov 01 13:25:08.105252 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.105251    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f347965-a09d-486c-bfa8-fa9cbe28e531" containerName="maas-api"
Nov 01 13:25:08.105425 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.105317    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f347965-a09d-486c-bfa8-fa9cbe28e531" containerName="maas-api"
Nov 01 13:25:08.105425 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.105328    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy"
Nov 01 13:25:08.109920 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.109905    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:08.157352 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.157311    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c687d10-ed98-4a64-9019-269a55aedbfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5\" (UID: \"5c687d10-ed98-4a64-9019-269a55aedbfb\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:08.157662 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.157644    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w27jc\" (UniqueName: \"kubernetes.io/projected/5c687d10-ed98-4a64-9019-269a55aedbfb-kube-api-access-w27jc\") pod \"payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5\" (UID: \"5c687d10-ed98-4a64-9019-269a55aedbfb\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:08.161168 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.161100    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-624dd4\""
Nov 01 13:25:08.207673 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.207644    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-624dd4-dockercfg-mfqw2\""
Nov 01 13:25:08.209607 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.209584    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"]
Nov 01 13:25:08.258556 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.258516    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w27jc\" (UniqueName: \"kubernetes.io/projected/5c687d10-ed98-4a64-9019-269a55aedbfb-kube-api-access-w27jc\") pod \"payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5\" (UID: \"5c687d10-ed98-4a64-9019-269a55aedbfb\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:08.258734 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.258652    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c687d10-ed98-4a64-9019-269a55aedbfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5\" (UID: \"5c687d10-ed98-4a64-9019-269a55aedbfb\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:08.259387 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.259368    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c687d10-ed98-4a64-9019-269a55aedbfb-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5\" (UID: \"5c687d10-ed98-4a64-9019-269a55aedbfb\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:08.272530 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.272494    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"]
Nov 01 13:25:08.273076 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.273062    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="f51f9dbb-0870-4cb1-8ddc-d440a86ee112" containerName="istio-proxy"
Nov 01 13:25:08.276434 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.276419    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:08.312501 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.312468    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w27jc\" (UniqueName: \"kubernetes.io/projected/5c687d10-ed98-4a64-9019-269a55aedbfb-kube-api-access-w27jc\") pod \"payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5\" (UID: \"5c687d10-ed98-4a64-9019-269a55aedbfb\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:08.329510 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.329482    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"]
Nov 01 13:25:08.359504 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.359481    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ffc82c-f64b-4041-afd8-7395b52a2c5b-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6\" (UID: \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:08.359649 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.359563    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb6bc\" (UniqueName: \"kubernetes.io/projected/75ffc82c-f64b-4041-afd8-7395b52a2c5b-kube-api-access-xb6bc\") pod \"payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6\" (UID: \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:08.422771 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.422726    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:08.461124 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.461082    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb6bc\" (UniqueName: \"kubernetes.io/projected/75ffc82c-f64b-4041-afd8-7395b52a2c5b-kube-api-access-xb6bc\") pod \"payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6\" (UID: \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:08.461858 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.461825    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ffc82c-f64b-4041-afd8-7395b52a2c5b-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6\" (UID: \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:08.462569 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.462543    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ffc82c-f64b-4041-afd8-7395b52a2c5b-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6\" (UID: \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:08.516141 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.516109    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb6bc\" (UniqueName: \"kubernetes.io/projected/75ffc82c-f64b-4041-afd8-7395b52a2c5b-kube-api-access-xb6bc\") pod \"payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6\" (UID: \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:08.589983 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.589949    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:08.786040 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.786016    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"]
Nov 01 13:25:08.788142 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:25:08.788114    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c687d10_ed98_4a64_9019_269a55aedbfb.slice/crio-1e8636681bd613a9c4b4c8c6fe34942da6da1b5db305748963a8411f26c62587 WatchSource:0}: Error finding container 1e8636681bd613a9c4b4c8c6fe34942da6da1b5db305748963a8411f26c62587: Status 404 returned error can't find the container with id 1e8636681bd613a9c4b4c8c6fe34942da6da1b5db305748963a8411f26c62587
Nov 01 13:25:08.849877 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.849847    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5" event={"ID":"5c687d10-ed98-4a64-9019-269a55aedbfb","Type":"ContainerStarted","Data":"1e8636681bd613a9c4b4c8c6fe34942da6da1b5db305748963a8411f26c62587"}
Nov 01 13:25:08.954068 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:08.954044    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"]
Nov 01 13:25:08.955660 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:25:08.955633    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75ffc82c_f64b_4041_afd8_7395b52a2c5b.slice/crio-ab6aa1560280a68fa9425026f53024a7bb26898b98e15c2e4e6cac2b47df41df WatchSource:0}: Error finding container ab6aa1560280a68fa9425026f53024a7bb26898b98e15c2e4e6cac2b47df41df: Status 404 returned error can't find the container with id ab6aa1560280a68fa9425026f53024a7bb26898b98e15c2e4e6cac2b47df41df
Nov 01 13:25:09.856406 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:09.856365    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5" event={"ID":"5c687d10-ed98-4a64-9019-269a55aedbfb","Type":"ContainerStarted","Data":"c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13"}
Nov 01 13:25:09.856905 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:09.856552    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:09.857895 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:09.857868    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6" event={"ID":"75ffc82c-f64b-4041-afd8-7395b52a2c5b","Type":"ContainerStarted","Data":"8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db"}
Nov 01 13:25:09.858040 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:09.857913    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6" event={"ID":"75ffc82c-f64b-4041-afd8-7395b52a2c5b","Type":"ContainerStarted","Data":"ab6aa1560280a68fa9425026f53024a7bb26898b98e15c2e4e6cac2b47df41df"}
Nov 01 13:25:09.858040 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:09.857956    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:09.965376 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:09.965312    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5" podStartSLOduration=1.965297651 podStartE2EDuration="1.965297651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:25:09.96352958 +0000 UTC m=+3769.881325246" watchObservedRunningTime="2026-11-01 13:25:09.965297651 +0000 UTC m=+3769.883093319"
Nov 01 13:25:10.074363 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:10.073446    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6" podStartSLOduration=2.073429108 podStartE2EDuration="2.073429108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:25:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:25:10.07234337 +0000 UTC m=+3769.990139046" watchObservedRunningTime="2026-11-01 13:25:10.073429108 +0000 UTC m=+3769.991224774"
Nov 01 13:25:11.000281 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.000247    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"]
Nov 01 13:25:11.004234 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.004211    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.053784 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.053756    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-624dd4-serving-cert\""
Nov 01 13:25:11.089458 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.089424    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvbj5\" (UniqueName: \"kubernetes.io/projected/3d8292fd-a091-458f-b699-607c2203e1fc-kube-api-access-tvbj5\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.089612 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.089470    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.089612 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.089591    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.089739 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.089664    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8292fd-a091-458f-b699-607c2203e1fc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.103071 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.103046    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"]
Nov 01 13:25:11.190412 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.190382    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.190613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.190437    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8292fd-a091-458f-b699-607c2203e1fc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.190613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.190504    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvbj5\" (UniqueName: \"kubernetes.io/projected/3d8292fd-a091-458f-b699-607c2203e1fc-kube-api-access-tvbj5\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.190613 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.190534    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.190736 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:25:11.190639    4095 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-serving-cert: secret "maas-api-e2e-authcfg-http-624dd4-serving-cert" not found
Nov 01 13:25:11.190736 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:25:11.190708    4095 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls podName:3d8292fd-a091-458f-b699-607c2203e1fc nodeName:}" failed. No retries permitted until 2026-11-01 13:25:11.690688138 +0000 UTC m=+3771.608483802 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls") pod "maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf" (UID: "3d8292fd-a091-458f-b699-607c2203e1fc") : secret "maas-api-e2e-authcfg-http-624dd4-serving-cert" not found
Nov 01 13:25:11.191232 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.191214    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8292fd-a091-458f-b699-607c2203e1fc-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.192828 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.192810    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-metrics-certs\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.261248 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.261176    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvbj5\" (UniqueName: \"kubernetes.io/projected/3d8292fd-a091-458f-b699-607c2203e1fc-kube-api-access-tvbj5\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.694672 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.694638    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.697045 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.697012    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:11.917376 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:11.917328    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:12.284583 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:12.284560    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"]
Nov 01 13:25:12.286304 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:25:12.286277    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8292fd_a091_458f_b699_607c2203e1fc.slice/crio-447e0d8c0914b9d81d9f1c29af7dc2853acdfcf0448c28e17f8008ca96a430b1 WatchSource:0}: Error finding container 447e0d8c0914b9d81d9f1c29af7dc2853acdfcf0448c28e17f8008ca96a430b1: Status 404 returned error can't find the container with id 447e0d8c0914b9d81d9f1c29af7dc2853acdfcf0448c28e17f8008ca96a430b1
Nov 01 13:25:12.873130 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:12.873095    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf" event={"ID":"3d8292fd-a091-458f-b699-607c2203e1fc","Type":"ContainerStarted","Data":"447e0d8c0914b9d81d9f1c29af7dc2853acdfcf0448c28e17f8008ca96a430b1"}
Nov 01 13:25:13.879120 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:13.879088    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf" event={"ID":"3d8292fd-a091-458f-b699-607c2203e1fc","Type":"ContainerStarted","Data":"bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026"}
Nov 01 13:25:13.879509 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:13.879167    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:13.990755 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:13.990704    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf" podStartSLOduration=3.088310398 podStartE2EDuration="3.990689734s" totalImagesPullingTime="902.379336ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-01 13:25:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:25:13.989452719 +0000 UTC m=+3773.907248386" watchObservedRunningTime="2026-11-01 13:25:13.990689734 +0000 UTC m=+3773.908485399"
Nov 01 13:25:19.128613 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:25:19.128570    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:25:19.890452 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:19.890423    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:25:20.865170 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:20.865143    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:25:20.865642 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:25:20.865208    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:25:49.113736 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:25:49.113697    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:26:19.113875 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:26:19.113832    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:26:49.112811 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:26:49.112772    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:27:03.890644 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:03.890602    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"]
Nov 01 13:27:03.989983 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:03.989940    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"]
Nov 01 13:27:08.891731 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:08.891693    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6" podUID="75ffc82c-f64b-4041-afd8-7395b52a2c5b" containerName="payload-processing" containerID="cri-o://8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db" gracePeriod=25
Nov 01 13:27:08.990659 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:08.990614    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5" podUID="5c687d10-ed98-4a64-9019-269a55aedbfb" containerName="payload-pre-processing" containerID="cri-o://c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13" gracePeriod=25
Nov 01 13:27:09.165641 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.165569    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:27:09.259692 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.259668    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:27:09.300167 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.300130    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c687d10-ed98-4a64-9019-269a55aedbfb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c687d10-ed98-4a64-9019-269a55aedbfb-plugins-config-volume\") pod \"5c687d10-ed98-4a64-9019-269a55aedbfb\" (UID: \"5c687d10-ed98-4a64-9019-269a55aedbfb\") "
Nov 01 13:27:09.300350 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.300192    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c687d10-ed98-4a64-9019-269a55aedbfb-kube-api-access-w27jc\" (UniqueName: \"kubernetes.io/projected/5c687d10-ed98-4a64-9019-269a55aedbfb-kube-api-access-w27jc\") pod \"5c687d10-ed98-4a64-9019-269a55aedbfb\" (UID: \"5c687d10-ed98-4a64-9019-269a55aedbfb\") "
Nov 01 13:27:09.300350 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.300241    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75ffc82c-f64b-4041-afd8-7395b52a2c5b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ffc82c-f64b-4041-afd8-7395b52a2c5b-plugins-config-volume\") pod \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\" (UID: \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\") "
Nov 01 13:27:09.300350 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.300272    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75ffc82c-f64b-4041-afd8-7395b52a2c5b-kube-api-access-xb6bc\" (UniqueName: \"kubernetes.io/projected/75ffc82c-f64b-4041-afd8-7395b52a2c5b-kube-api-access-xb6bc\") pod \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\" (UID: \"75ffc82c-f64b-4041-afd8-7395b52a2c5b\") "
Nov 01 13:27:09.300571 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.300541    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c687d10-ed98-4a64-9019-269a55aedbfb-plugins-config-volume" pod "5c687d10-ed98-4a64-9019-269a55aedbfb" (UID: "5c687d10-ed98-4a64-9019-269a55aedbfb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:27:09.300667 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.300642    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75ffc82c-f64b-4041-afd8-7395b52a2c5b-plugins-config-volume" pod "75ffc82c-f64b-4041-afd8-7395b52a2c5b" (UID: "75ffc82c-f64b-4041-afd8-7395b52a2c5b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:27:09.300820 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.300793    4095 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5c687d10-ed98-4a64-9019-269a55aedbfb-plugins-config-volume\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:09.300820 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.300817    4095 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75ffc82c-f64b-4041-afd8-7395b52a2c5b-plugins-config-volume\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:09.302454 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.302430    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75ffc82c-f64b-4041-afd8-7395b52a2c5b-kube-api-access-xb6bc" pod "75ffc82c-f64b-4041-afd8-7395b52a2c5b" (UID: "75ffc82c-f64b-4041-afd8-7395b52a2c5b"). InnerVolumeSpecName "kube-api-access-xb6bc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:27:09.302557 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.302474    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c687d10-ed98-4a64-9019-269a55aedbfb-kube-api-access-w27jc" pod "5c687d10-ed98-4a64-9019-269a55aedbfb" (UID: "5c687d10-ed98-4a64-9019-269a55aedbfb"). InnerVolumeSpecName "kube-api-access-w27jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:27:09.401450 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.401421    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w27jc\" (UniqueName: \"kubernetes.io/projected/5c687d10-ed98-4a64-9019-269a55aedbfb-kube-api-access-w27jc\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:09.401450 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.401448    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb6bc\" (UniqueName: \"kubernetes.io/projected/75ffc82c-f64b-4041-afd8-7395b52a2c5b-kube-api-access-xb6bc\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:09.459652 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.459568    4095 generic.go:356] "Generic (PLEG): container finished" podID="5c687d10-ed98-4a64-9019-269a55aedbfb" containerID="c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13" exitCode=0
Nov 01 13:27:09.459652 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.459610    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"
Nov 01 13:27:09.459852 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.459656    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5" event={"ID":"5c687d10-ed98-4a64-9019-269a55aedbfb","Type":"ContainerDied","Data":"c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13"}
Nov 01 13:27:09.459852 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.459708    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5" event={"ID":"5c687d10-ed98-4a64-9019-269a55aedbfb","Type":"ContainerDied","Data":"1e8636681bd613a9c4b4c8c6fe34942da6da1b5db305748963a8411f26c62587"}
Nov 01 13:27:09.459852 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.459735    4095 scope.go:122] "RemoveContainer" containerID="c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13"
Nov 01 13:27:09.461125 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.461093    4095 generic.go:356] "Generic (PLEG): container finished" podID="75ffc82c-f64b-4041-afd8-7395b52a2c5b" containerID="8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db" exitCode=0
Nov 01 13:27:09.461238 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.461135    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"
Nov 01 13:27:09.461238 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.461156    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6" event={"ID":"75ffc82c-f64b-4041-afd8-7395b52a2c5b","Type":"ContainerDied","Data":"8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db"}
Nov 01 13:27:09.461238 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.461193    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6" event={"ID":"75ffc82c-f64b-4041-afd8-7395b52a2c5b","Type":"ContainerDied","Data":"ab6aa1560280a68fa9425026f53024a7bb26898b98e15c2e4e6cac2b47df41df"}
Nov 01 13:27:09.468873 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.468851    4095 scope.go:122] "RemoveContainer" containerID="c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13"
Nov 01 13:27:09.469161 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:27:09.469143    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13\": container with ID starting with c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13 not found: ID does not exist" containerID="c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13"
Nov 01 13:27:09.469241 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.469169    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13"} err="failed to get container status \"c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13\": rpc error: code = NotFound desc = could not find container \"c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13\": container with ID starting with c615a24b6187f08ab5ff9634657e11e6deaecc1ec639da8b99f1aed865a75e13 not found: ID does not exist"
Nov 01 13:27:09.469241 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.469184    4095 scope.go:122] "RemoveContainer" containerID="8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db"
Nov 01 13:27:09.479238 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.479219    4095 scope.go:122] "RemoveContainer" containerID="8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db"
Nov 01 13:27:09.479498 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:27:09.479478    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db\": container with ID starting with 8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db not found: ID does not exist" containerID="8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db"
Nov 01 13:27:09.479543 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.479506    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db"} err="failed to get container status \"8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db\": rpc error: code = NotFound desc = could not find container \"8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db\": container with ID starting with 8cf9350d373f5afd7ae9df86e39b957fce545d9909058522d21592cf63c936db not found: ID does not exist"
Nov 01 13:27:09.617488 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.617456    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"]
Nov 01 13:27:09.623424 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.623399    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-624dd4-65846dff-6ztj5"]
Nov 01 13:27:09.727728 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.727648    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"]
Nov 01 13:27:09.735469 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:09.735444    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-624dd4-649b9767f-qpxq6"]
Nov 01 13:27:10.188882 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:10.188847    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c687d10-ed98-4a64-9019-269a55aedbfb" path="/var/lib/kubelet/pods/5c687d10-ed98-4a64-9019-269a55aedbfb/volumes"
Nov 01 13:27:10.189323 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:10.189260    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75ffc82c-f64b-4041-afd8-7395b52a2c5b" path="/var/lib/kubelet/pods/75ffc82c-f64b-4041-afd8-7395b52a2c5b/volumes"
Nov 01 13:27:11.889451 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:11.889419    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"]
Nov 01 13:27:11.889890 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:11.889647    4095 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf" podUID="3d8292fd-a091-458f-b699-607c2203e1fc" containerName="maas-api" containerID="cri-o://bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026" gracePeriod=30
Nov 01 13:27:11.931156 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:27:11.931123    4095 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d8292fd_a091_458f_b699_607c2203e1fc.slice/crio-bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026.scope\": RecentStats: unable to find data in memory cache]"
Nov 01 13:27:12.144993 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.144925    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:27:12.228169 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.228145    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d8292fd-a091-458f-b699-607c2203e1fc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8292fd-a091-458f-b699-607c2203e1fc-openshift-service-ca\") pod \"3d8292fd-a091-458f-b699-607c2203e1fc\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") "
Nov 01 13:27:12.228438 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.228200    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-metrics-certs\") pod \"3d8292fd-a091-458f-b699-607c2203e1fc\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") "
Nov 01 13:27:12.228438 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.228285    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d8292fd-a091-458f-b699-607c2203e1fc-kube-api-access-tvbj5\" (UniqueName: \"kubernetes.io/projected/3d8292fd-a091-458f-b699-607c2203e1fc-kube-api-access-tvbj5\") pod \"3d8292fd-a091-458f-b699-607c2203e1fc\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") "
Nov 01 13:27:12.228438 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.228377    4095 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls\") pod \"3d8292fd-a091-458f-b699-607c2203e1fc\" (UID: \"3d8292fd-a091-458f-b699-607c2203e1fc\") "
Nov 01 13:27:12.228647 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.228526    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d8292fd-a091-458f-b699-607c2203e1fc-openshift-service-ca" pod "3d8292fd-a091-458f-b699-607c2203e1fc" (UID: "3d8292fd-a091-458f-b699-607c2203e1fc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 01 13:27:12.228778 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.228755    4095 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d8292fd-a091-458f-b699-607c2203e1fc-openshift-service-ca\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:12.230865 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.230831    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls" pod "3d8292fd-a091-458f-b699-607c2203e1fc" (UID: "3d8292fd-a091-458f-b699-607c2203e1fc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:27:12.231004 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.230957    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d8292fd-a091-458f-b699-607c2203e1fc-kube-api-access-tvbj5" pod "3d8292fd-a091-458f-b699-607c2203e1fc" (UID: "3d8292fd-a091-458f-b699-607c2203e1fc"). InnerVolumeSpecName "kube-api-access-tvbj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 01 13:27:12.231210 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.231188    4095 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-metrics-certs" pod "3d8292fd-a091-458f-b699-607c2203e1fc" (UID: "3d8292fd-a091-458f-b699-607c2203e1fc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 01 13:27:12.329330 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.329296    4095 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvbj5\" (UniqueName: \"kubernetes.io/projected/3d8292fd-a091-458f-b699-607c2203e1fc-kube-api-access-tvbj5\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:12.329330 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.329327    4095 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-maas-api-tls\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:12.329330 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.329338    4095 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d8292fd-a091-458f-b699-607c2203e1fc-metrics-certs\") on node \"ip-10-0-140-168.us-west-2.compute.internal\" DevicePath \"\""
Nov 01 13:27:12.478252 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.478171    4095 generic.go:356] "Generic (PLEG): container finished" podID="3d8292fd-a091-458f-b699-607c2203e1fc" containerID="bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026" exitCode=0
Nov 01 13:27:12.478252 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.478218    4095 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"
Nov 01 13:27:12.478445 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.478248    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf" event={"ID":"3d8292fd-a091-458f-b699-607c2203e1fc","Type":"ContainerDied","Data":"bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026"}
Nov 01 13:27:12.478445 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.478298    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf" event={"ID":"3d8292fd-a091-458f-b699-607c2203e1fc","Type":"ContainerDied","Data":"447e0d8c0914b9d81d9f1c29af7dc2853acdfcf0448c28e17f8008ca96a430b1"}
Nov 01 13:27:12.478445 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.478322    4095 scope.go:122] "RemoveContainer" containerID="bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026"
Nov 01 13:27:12.488263 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.488241    4095 scope.go:122] "RemoveContainer" containerID="bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026"
Nov 01 13:27:12.488517 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:27:12.488498    4095 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026\": container with ID starting with bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026 not found: ID does not exist" containerID="bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026"
Nov 01 13:27:12.488566 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.488526    4095 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026"} err="failed to get container status \"bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026\": rpc error: code = NotFound desc = could not find container \"bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026\": container with ID starting with bce365dad9b3e7aa332c06988c61d29b4c962d7d7f0d8f9cf6a3ece5d7843026 not found: ID does not exist"
Nov 01 13:27:12.633771 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.633736    4095 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"]
Nov 01 13:27:12.637041 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:12.637019    4095 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-624dd4-5ffc99db64-b7rlf"]
Nov 01 13:27:14.188575 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:14.188537    4095 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d8292fd-a091-458f-b699-607c2203e1fc" path="/var/lib/kubelet/pods/3d8292fd-a091-458f-b699-607c2203e1fc/volumes"
Nov 01 13:27:19.114058 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:27:19.114022    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:27:20.689963 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:20.689933    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:27:20.703580 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:27:20.703555    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hpzwg_c76d9907-a77b-455f-9437-660fd451b40b/ovn-acl-logging/0.log"
Nov 01 13:27:49.114363 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:27:49.114186    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:28:19.112348 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:28:19.112313    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:28:45.051639 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:45.051612    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-ktl4n_649e29f1-f8a8-490e-a0ac-a729006388be/server/0.log"
Nov 01 13:28:46.268837 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:46.268804    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6985888875-4bv96_4d6b5e9a-f783-45b8-8a1f-d88ac3810de7/maas-api/0.log"
Nov 01 13:28:49.113356 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:28:49.113323    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:28:49.789187 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:49.789154    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nhgx4_8bf1a5f6-441b-45cd-a923-74d51fcaa90b/kuadrant-console-plugin/0.log"
Nov 01 13:28:51.729683 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:51.729644    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf7dc98fb-4t6bt_5c54b04d-ee14-4f0c-a7ff-0cba080217a2/kube-auth-proxy/0.log"
Nov 01 13:28:52.161700 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:52.161670    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-29nw6_7352735c-f1a9-4480-bca7-c9cbe3f9e8e2/istio-proxy/0.log"
Nov 01 13:28:52.596662 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:52.596592    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-9m98v_f3566191-2934-4500-90e2-c12dbffb0354/istio-proxy/0.log"
Nov 01 13:28:52.820855 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:52.820823    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-b4vmw_7adcf7b8-dcff-4e54-9918-98dec1cf66b4/istio-proxy/0.log"
Nov 01 13:28:53.248912 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:53.248878    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-gjsds_490cb0f8-5280-4505-9141-43cd22eb0cac/istio-proxy/0.log"
Nov 01 13:28:55.690085 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:55.690048    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7_6bee1b64-413a-4f04-b05a-b7ad5acda1a2/main/0.log"
Nov 01 13:28:56.121927 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:56.121901    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4r49f_e6bf57e8-4c6b-4a61-ad73-650f4d6a8696/main/0.log"
Nov 01 13:28:56.535856 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:56.535771    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh_68b51645-28d7-42ff-8aa1-0e4d49c02444/main/0.log"
Nov 01 13:28:56.961959 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:28:56.961924    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf_9086be84-9c9e-4501-82ca-a580e08b3fd0/main/0.log"
Nov 01 13:29:08.029573 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:08.029535    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-ktl4n_649e29f1-f8a8-490e-a0ac-a729006388be/server/0.log"
Nov 01 13:29:09.027621 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:09.027584    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6985888875-4bv96_4d6b5e9a-f783-45b8-8a1f-d88ac3810de7/maas-api/0.log"
Nov 01 13:29:11.980429 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:11.980400    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nhgx4_8bf1a5f6-441b-45cd-a923-74d51fcaa90b/kuadrant-console-plugin/0.log"
Nov 01 13:29:13.689082 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:13.689044    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7bf7dc98fb-4t6bt_5c54b04d-ee14-4f0c-a7ff-0cba080217a2/kube-auth-proxy/0.log"
Nov 01 13:29:14.121118 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:14.121084    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-29nw6_7352735c-f1a9-4480-bca7-c9cbe3f9e8e2/istio-proxy/0.log"
Nov 01 13:29:14.553136 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:14.553047    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-9m98v_f3566191-2934-4500-90e2-c12dbffb0354/istio-proxy/0.log"
Nov 01 13:29:14.751789 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:14.751759    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-b4vmw_7adcf7b8-dcff-4e54-9918-98dec1cf66b4/istio-proxy/0.log"
Nov 01 13:29:15.184127 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:15.184094    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-7df854bf67-gjsds_490cb0f8-5280-4505-9141-43cd22eb0cac/istio-proxy/0.log"
Nov 01 13:29:17.377359 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:17.377324    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-mwpp7_6bee1b64-413a-4f04-b05a-b7ad5acda1a2/main/0.log"
Nov 01 13:29:17.781889 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:17.781807    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-4r49f_e6bf57e8-4c6b-4a61-ad73-650f4d6a8696/main/0.log"
Nov 01 13:29:18.212468 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:18.212437    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8k7plh_68b51645-28d7-42ff-8aa1-0e4d49c02444/main/0.log"
Nov 01 13:29:18.639495 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:29:18.639468    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-b4jmf_9086be84-9c9e-4501-82ca-a580e08b3fd0/main/0.log"
Nov 01 13:29:19.113470 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:29:19.113434    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:29:49.112958 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:29:49.112917    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:30:19.113132 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:30:19.113096    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:30:33.244118 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:33.244043    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5pg82_3a7a84c2-0167-4f45-82b8-d8da5f192fac/global-pull-secret-syncer/0.log"
Nov 01 13:30:37.098881 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:37.098842    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-nsn56_a718c0bb-3ac8-432b-9b69-60e61375994f/konnectivity-agent/0.log"
Nov 01 13:30:37.881040 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:37.881006    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-140-168.us-west-2.compute.internal_7042c7ca4bb39d530e5b25fb56952b15/haproxy/0.log"
Nov 01 13:30:43.334450 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.334411    4095 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"]
Nov 01 13:30:43.335022 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.334990    4095 cpu_manager.go:414] "containerMap: removing container" podUID="5c687d10-ed98-4a64-9019-269a55aedbfb" containerName="payload-pre-processing"
Nov 01 13:30:43.335022 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.335009    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c687d10-ed98-4a64-9019-269a55aedbfb" containerName="payload-pre-processing"
Nov 01 13:30:43.335215 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.335031    4095 cpu_manager.go:414] "containerMap: removing container" podUID="3d8292fd-a091-458f-b699-607c2203e1fc" containerName="maas-api"
Nov 01 13:30:43.335215 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.335038    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d8292fd-a091-458f-b699-607c2203e1fc" containerName="maas-api"
Nov 01 13:30:43.335215 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.335086    4095 cpu_manager.go:414] "containerMap: removing container" podUID="75ffc82c-f64b-4041-afd8-7395b52a2c5b" containerName="payload-processing"
Nov 01 13:30:43.335215 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.335094    4095 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75ffc82c-f64b-4041-afd8-7395b52a2c5b" containerName="payload-processing"
Nov 01 13:30:43.335215 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.335195    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c687d10-ed98-4a64-9019-269a55aedbfb" containerName="payload-pre-processing"
Nov 01 13:30:43.335215 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.335205    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d8292fd-a091-458f-b699-607c2203e1fc" containerName="maas-api"
Nov 01 13:30:43.335418 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.335217    4095 memory_manager.go:365] "RemoveStaleState removing state" podUID="75ffc82c-f64b-4041-afd8-7395b52a2c5b" containerName="payload-processing"
Nov 01 13:30:43.338470 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.338454    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.387748 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.387723    4095 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9plmq\"/\"default-dockercfg-k7gkk\""
Nov 01 13:30:43.434858 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.434832    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9plmq\"/\"openshift-service-ca.crt\""
Nov 01 13:30:43.435035 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.434925    4095 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9plmq\"/\"kube-root-ca.crt\""
Nov 01 13:30:43.436777 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.436758    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"]
Nov 01 13:30:43.447864 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.447839    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-proc\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.448086 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.447901    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf6k4\" (UniqueName: \"kubernetes.io/projected/39954323-8fcb-41bd-9de7-94d0c68b8964-kube-api-access-bf6k4\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.448168 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.448077    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-sys\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.448168 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.448113    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-podres\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.448257 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.448212    4095 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-lib-modules\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.548826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.548785    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-sys\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.548826 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.548826    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-podres\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.549116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.548875    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-lib-modules\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.549116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.548909    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-sys\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.549116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.548988    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-proc\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.549116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.549055    4095 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf6k4\" (UniqueName: \"kubernetes.io/projected/39954323-8fcb-41bd-9de7-94d0c68b8964-kube-api-access-bf6k4\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.549116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.549055    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-podres\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.549116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.549069    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-lib-modules\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.549116 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.549055    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/39954323-8fcb-41bd-9de7-94d0c68b8964-proc\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.602682 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.602608    4095 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf6k4\" (UniqueName: \"kubernetes.io/projected/39954323-8fcb-41bd-9de7-94d0c68b8964-kube-api-access-bf6k4\") pod \"perf-node-gather-daemonset-ssx9f\" (UID: \"39954323-8fcb-41bd-9de7-94d0c68b8964\") " pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.651196 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.651167    4095 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:43.763433 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:43.763407    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-nhgx4_8bf1a5f6-441b-45cd-a923-74d51fcaa90b/kuadrant-console-plugin/0.log"
Nov 01 13:30:44.012266 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:44.012234    4095 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"]
Nov 01 13:30:44.014554 ip-10-0-140-168 kubenswrapper[4095]: W1101 13:30:44.014527    4095 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod39954323_8fcb_41bd_9de7_94d0c68b8964.slice/crio-0a947fbe7e89dee9fb4abaf5f96349fd583b549798baffe0685d85cb51bf70b0 WatchSource:0}: Error finding container 0a947fbe7e89dee9fb4abaf5f96349fd583b549798baffe0685d85cb51bf70b0: Status 404 returned error can't find the container with id 0a947fbe7e89dee9fb4abaf5f96349fd583b549798baffe0685d85cb51bf70b0
Nov 01 13:30:44.716167 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:44.716132    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f" event={"ID":"39954323-8fcb-41bd-9de7-94d0c68b8964","Type":"ContainerStarted","Data":"d6968b5ca6e687fa67efb9171fd340afc4a5e6443ccda682dc2e0961600f6d5a"}
Nov 01 13:30:44.716627 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:44.716182    4095 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f" event={"ID":"39954323-8fcb-41bd-9de7-94d0c68b8964","Type":"ContainerStarted","Data":"0a947fbe7e89dee9fb4abaf5f96349fd583b549798baffe0685d85cb51bf70b0"}
Nov 01 13:30:44.716627 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:44.716274    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:44.823659 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:44.823604    4095 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f" podStartSLOduration=1.8235891309999999 podStartE2EDuration="1.823589131s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-01 13:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-01 13:30:44.823239191 +0000 UTC m=+4104.741034856" watchObservedRunningTime="2026-11-01 13:30:44.823589131 +0000 UTC m=+4104.741384835"
Nov 01 13:30:47.704813 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:47.704783    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9cd7ab-b91c-433f-bca3-6889327fa42c/alertmanager/0.log"
Nov 01 13:30:48.110270 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:48.110237    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9cd7ab-b91c-433f-bca3-6889327fa42c/config-reloader/0.log"
Nov 01 13:30:48.516078 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:48.515988    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9cd7ab-b91c-433f-bca3-6889327fa42c/kube-rbac-proxy-web/0.log"
Nov 01 13:30:48.920583 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:48.920559    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9cd7ab-b91c-433f-bca3-6889327fa42c/kube-rbac-proxy/0.log"
Nov 01 13:30:49.114305 ip-10-0-140-168 kubenswrapper[4095]: E1101 13:30:49.114265    4095 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-rxlpj" containerName="extractor"
Nov 01 13:30:49.326331 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:49.326255    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9cd7ab-b91c-433f-bca3-6889327fa42c/kube-rbac-proxy-metric/0.log"
Nov 01 13:30:49.733575 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:49.733542    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9cd7ab-b91c-433f-bca3-6889327fa42c/prom-label-proxy/0.log"
Nov 01 13:30:50.142506 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:50.142477    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_fa9cd7ab-b91c-433f-bca3-6889327fa42c/init-config-reloader/0.log"
Nov 01 13:30:50.732277 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:50.732248    4095 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9plmq/perf-node-gather-daemonset-ssx9f"
Nov 01 13:30:50.970231 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:50.970201    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jkjf8_7b5a5120-e40c-410c-9f68-93571e0467a6/kube-state-metrics/0.log"
Nov 01 13:30:51.367494 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:51.367465    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jkjf8_7b5a5120-e40c-410c-9f68-93571e0467a6/kube-rbac-proxy-main/0.log"
Nov 01 13:30:51.774376 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:51.774291    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-jkjf8_7b5a5120-e40c-410c-9f68-93571e0467a6/kube-rbac-proxy-self/0.log"
Nov 01 13:30:52.184222 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:52.184194    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-6d87d6bbc7-df484_016791b0-0300-4229-a88e-2446533ce831/metrics-server/0.log"
Nov 01 13:30:54.228308 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:54.228277    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hj2v5_d3076378-1092-4108-90af-c56ce923a846/node-exporter/0.log"
Nov 01 13:30:54.632089 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:54.632055    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hj2v5_d3076378-1092-4108-90af-c56ce923a846/kube-rbac-proxy/0.log"
Nov 01 13:30:55.038447 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:55.038365    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-hj2v5_d3076378-1092-4108-90af-c56ce923a846/init-textfile/0.log"
Nov 01 13:30:57.903625 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:57.903593    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cccc1506-3c47-4a60-995b-557c7ed41a65/prometheus/0.log"
Nov 01 13:30:58.312480 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:58.312403    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cccc1506-3c47-4a60-995b-557c7ed41a65/config-reloader/0.log"
Nov 01 13:30:58.718583 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:58.718557    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cccc1506-3c47-4a60-995b-557c7ed41a65/thanos-sidecar/0.log"
Nov 01 13:30:59.124334 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:59.124308    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cccc1506-3c47-4a60-995b-557c7ed41a65/kube-rbac-proxy-web/0.log"
Nov 01 13:30:59.530191 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:59.530110    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cccc1506-3c47-4a60-995b-557c7ed41a65/kube-rbac-proxy/0.log"
Nov 01 13:30:59.935184 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:30:59.935152    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cccc1506-3c47-4a60-995b-557c7ed41a65/kube-rbac-proxy-thanos/0.log"
Nov 01 13:31:00.342225 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:00.342142    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_cccc1506-3c47-4a60-995b-557c7ed41a65/init-config-reloader/0.log"
Nov 01 13:31:00.756356 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:00.756325    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8tt9c_3a6cb753-afaf-4be7-b341-cb0da3518da0/prometheus-operator/0.log"
Nov 01 13:31:01.156183 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:01.156145    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-9cd89b74c-8tt9c_3a6cb753-afaf-4be7-b341-cb0da3518da0/kube-rbac-proxy/0.log"
Nov 01 13:31:01.565594 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:01.565512    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-2f6kw_2de24e27-6407-4189-9fda-c8468c9bef28/prometheus-operator-admission-webhook/0.log"
Nov 01 13:31:01.978760 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:01.978735    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c749d9f6b-86ntp_4289e74e-8746-4732-9ffd-70b4e7c746c4/telemeter-client/0.log"
Nov 01 13:31:02.385607 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:02.385578    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c749d9f6b-86ntp_4289e74e-8746-4732-9ffd-70b4e7c746c4/reload/0.log"
Nov 01 13:31:02.792175 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:02.792081    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c749d9f6b-86ntp_4289e74e-8746-4732-9ffd-70b4e7c746c4/kube-rbac-proxy/0.log"
Nov 01 13:31:03.203697 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:03.203665    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c896dfbdb-ldfhk_9bff35e5-ff1f-4dfd-b05d-f87d37d9011e/thanos-query/0.log"
Nov 01 13:31:03.611759 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:03.611732    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c896dfbdb-ldfhk_9bff35e5-ff1f-4dfd-b05d-f87d37d9011e/kube-rbac-proxy-web/0.log"
Nov 01 13:31:04.017785 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:04.017677    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c896dfbdb-ldfhk_9bff35e5-ff1f-4dfd-b05d-f87d37d9011e/kube-rbac-proxy/0.log"
Nov 01 13:31:04.423290 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:04.423261    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c896dfbdb-ldfhk_9bff35e5-ff1f-4dfd-b05d-f87d37d9011e/prom-label-proxy/0.log"
Nov 01 13:31:04.829889 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:04.829816    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c896dfbdb-ldfhk_9bff35e5-ff1f-4dfd-b05d-f87d37d9011e/kube-rbac-proxy-rules/0.log"
Nov 01 13:31:05.237934 ip-10-0-140-168 kubenswrapper[4095]: I1101 13:31:05.237901    4095 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7c896dfbdb-ldfhk_9bff35e5-ff1f-4dfd-b05d-f87d37d9011e/kube-rbac-proxy-metrics/0.log"